trying later

aemw

@@ -0,0 +1 @@
+[{"id_text": "P500443", "project_name": "aemw/alalakh/idrimi", "raw_text": "I am Idrimi, the son of Il\u012b-ilimma, a servant of IM, Hebat and I\u0160TAR, the lady of Alalah, the lady, my lady. In Aleppo, the house of my father, a bad thing occurred, so we fled to the Emarites, sisters [o]f my mother, and settled at Emar.  Though my older brothers lived with me, none deliberated matters as I did.   I thought like this: \u201cWho indeed ... at his father\u2019s house, but who is nonetheless a servant among the people of Emar?\u201d  I took my horse, my chariot, and my chariot driver.  I crossed over into a desolate region, where I joined the Sutean pastoralists.  With them(text: him), to ....  I spent the night. The next day, I set out and journeyed to the land of Canaan.  The city of Ammiya sits in the land of Canaan, (and) in Ammiya dwell people originally from Aleppo (and) the lands of Muki\u0161, Niye, and Ama\u02bee.  They recognized me \u2013 that I was the son of their lord \u2013 and gathered around me, and in that way, I was elevated in rank.  For seven long years, I stayed among the habiru. I released birds (as auguries), I inspected (the livers of) lambs, and seven years (later), the Storm God turned toward me. I built ships.  I loaded soldiers onto the ships, I approached the land of Muki\u0161 by sea, and I reached dry land in front of Mount Hazzi (i.e., Mount Cassius/Jebel Aqraa).  I went up (the mountain), and when my land heard me, they brought oxen and sheep to me.  In one day, as one man, the lands of Niya, Ama\u02bee, (and) Muki\u0161 and the city of Alalah, my city, turned to me. My brothers heard (about this) and came to me.  My brothers toiled for me, (and) I protected my brothers. Furthermore, for seven years, Parattarna, the powerful king, the king of the Hurrians, made enemies for me.  In the seventh of the years, I sent a message to Parattarna, the king, king of the Umman-manda and told of the service of my forefathers \u2013 that my forefathers toiled for them so that our words were for the kings of the Hurrians.  It was good.  They established a powerful oath between them.  The powerful king listened to the service of our ancestors and the oath that was between them, and he respected the \u201cmark\u201d of the oath.  He accepted my greeting gift because of the matter of the oath and because of our service.  I enlarged ... for (lit. of) a sacrifice, and so I returned a lost house to him:  I seized a previously-abandoned hem for him oath in my status as a loyal retainer, and then I was king for Alalah. Kings to my right and my left came up here to me, and thus I was made equal to them. Because the city wall of the forefathers lay tumbled on the ground, I caused (bricks) to rise up from on the ground, and I set them (bricks?) up high as the sky I took troops, went up to the land of Hatti, and captured seven cities:  Pa\u0161\u0161ahe, Damarutla, Huluhhan, Zila, I\u02bee, Uluzila, and Zaruna.  I captured these cities, and I destroyed others.  The land of Hatti did not gather and come against me.  I did what I wished.  I carried off captives and took all types of goods and possessions.  I distributed (the plunder) to my auxiliaries, my brothers, and my allies.  I myself took their weapon(s), though. I returned to the land of Muki\u0161 and entered Alalah, my city.  With the captives and livestock, with all the goods and possessions that I brought down from the land of Hatti, I had a house built.  I made my throne like the thrones of kings.  I made my brothers like the brothers of kings, my sons like their sons, and my companions like their companions. As for the inhabitants whose dwelling was (already) in my land, in kindness, I caused them to reside.  I housed even those who lacked a dwelling.  I stabilized my land and made my cities like our previous ones. And just as our father established the signs of the gods of Alalah, I was regularly performing the offerings (Akk. gloss: the offerings) for our grandfather that he regularly caused to be performed.  I regularly performed these (rites), and I entrusted them to IM-nerari, my son. Whoever removes this statue of mine \u2013 may Heaven curse him, (and) the Underworld gather up his offspring.  May the gods of heaven and earth measure out his reign and his land for him.  Whoever alters it (the inscription?) or erases <it> \u2013 may IM, the lord of heaven and earth (Akk. gloss: earth), and the great gods, destroy his name and his seed from his land. \u0160arruwa, the scribe, is a servant of the Storm God, Sun God, Moon God, and I\u0160TAR.  \u0160arruwa is the scribe who wrote (the inscription on) this statue. May the gods of heaven and earth keep him alive (and) protect him.  May they be good to him.  May the Sun God, lord of above and below, lord of ghosts, be the one who revives him. I was king for 30 years.  I wrote my labors on myself.  May it encourage them (the descendants) so that they (the descendants) may pray to me regularlLine truncated

akklove

@@ -0,0 +1 @@
+[{"id_text": "P251711", "project_name": "akklove", "raw_text": "A woman she is! May they not talk favorably with e[ach other(?)]! May they not [kis]s(?) [each other] in the course of night! May they not tell each other whatever is pure in their hearts! Oh exalted one, great daughter of S\u00een! The mistress of the en, you are superior: who can be seen next to you? A lady through true ordinances, the great one of the lord, how majestic you are! [Conceived(?)] from a pure heart, you surpass your bearing mother! The one(m.) that makes the field flourish... the pure ... they will s[ing(?)].  The true goddess of ordinances, greatly superior, the head of the noble ones."}, {"id_text": "P251898", "project_name": "akklove", "raw_text": "Daughter of an exile \u2013 my heart rejoiced; (she) took away the laugh of (my) mother. My mood plunged into the infesting heart \u2013 her lovemaking is an infesting love. She is sweet as honey, She is fresh like wine to the nose. (My) mood is (fixed on her) fruits. As life, mentioning her name is insatiable, Though she is distant, her merriment sets in motion. 14 [One who has] a loved-one chases (lit. sees) a dream: my heart [is broken], my mood is agitated! [Had] I\u0161tar schemed this affair, lovers could not [ever] be seen! My he[art] rejoiced! My mood became cheerful! ... is the truth. Restless and constantly trembling: I (Text: he) awoke to the noise of swallows, writhed in (my) bedroom on the bed: The lov[e-charm] returned into m[y hear]t!"}, {"id_text": "P252006", "project_name": "akklove", "raw_text": "[Keep] her apart! keep [her] apart! [The] daughters of Anu, the lights of heaven, [in day-ti]me(?) purified the sky of Anu. Love came about, twittering over the people; May Love twitter over me! Let me cast (a spell), let me speak, let me utter words of love (lit.: let me speak, let me love): \u201cThink of me as an a\u0161nugallum-snake! May your face rejoice as a wild cow! Do not wait on your father\u2019s counsel, Do not heed your mother\u2019s advice!\u201d If she is a hierodule may she [...] her support; If she is a cloister-lady may she [...] her gift; If she is a harlot may she clean(?) her tavern, may she throw herself at me! The incantation is not mine: (it is) the incantation (which) Ea and I\u0161tar have created."}, {"id_text": "P252071", "project_name": "akklove", "raw_text": "I have torn the thorn, I will be sowing a vine! I have poured water onto the fierce fire. Love me as your lamb, Encircle me as your small cattle. Look at me!"}, {"id_text": "P252226", "project_name": "akklove", "raw_text": "I scorn the girl who does not adore (me), I have no desire for the girl who does not play up to (me), I shall not give her (my) allures! I shall stand over her! Talking in order to disagree - why does it exist? Shall I give my love where the heart of darkness is? No one can control [me/it]! Break off, leave! You ma[ke me speech]less, (there is) not much to talk (about)! What I said is said and [...] I will not ch[ange it for y]ou! The one who [lies down] for a woman, is (nothing but) a weevil of the wall! If he does not th[rive], he is no match for a man. You(f.) were born the daughter of a substitute, with no dowry! You have a mole on (your) forehead! As long as you show no respect, pu[tting yourself] to shame, I shall tell you where your (right) place is! You do not listen to me, you. (By) following your heart, mounting the clouds, You keep chasing lovers away! This goes over (the limit)! Where is the source of your rebellion? Ask the previous women! (You are) like a field of salt, (can I) be happy with all (this)? I was happy with the fruit, (can I) be happy with all (this)? and [...] mouth [...] The loved one [...] You should not put [...] To your canal \u2013 no one will come near it! Your lord, your task \u2013 do not place (them) in the salt! your field is all too well known! The (fact) that you did not bring for me (good) news from your womb, as a baby of men \u2013 should I swallow (that) potsherd? I will release the bitch! One who bolts down a stone in order to release you when would he have his word? Verily, you, when (somebody) came near [you], you were convulsing like the goddess Belili, you were dancing at dawn (instead of) sleeping. You are imposing grief on yourself!"}, {"id_text": "P252332", "project_name": "akklove", "raw_text": "\u201cStand(f.) in the light of the windows!\u201d \u201cIf I fall asleep \u2013 you(f.), wake me up!\u201d \u201cI am going \u2013 move(m.) on! Do not rebel!\u201d \u201cMy restless girl was walking to and fro like a hoopoe.\u201d \u201cBig one(m.), big one, do not arise!\u201d \u201cLet me look at you(m.) by the light of the windows!\u201d \u201cLet us complete the deed of lovemaking!\u201d \u201cLet me grow long for the girl!\u201d \u201cIt is so enlarged! That of an elephant was smaller than yours!\u201d \u201cLet us invite the ...! She is taken by force regarding her vulva\u201d \u201cAsk the canal-inspector, the god Line truncated

amgg

@@ -0,0 +1 @@
+[]

jsons_unzipped

@@ -0,0 +1 @@
+C:/Users/Saret/Neutral Folder/dh/

project_list

@@ -1,7 +1,7 @@
-adsd
-aemw
-akklove
-amgg
+#adsd
+#aemw
+#akklove
+#amgg
 ario
 armep
 arrim

project_notebook.ipynb

@@ -10,7 +10,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 30,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -31,7 +31,20 @@
    "cell_type": "code",
    "execution_count": 17,
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "ename": "TypeError",
+     "evalue": "'NoneType' object is not subscriptable",
+     "output_type": "error",
+     "traceback": [
+      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
+      "Cell \u001b[1;32mIn[17], line 5\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[39mwith\u001b[39;00m \u001b[39mopen\u001b[39m(\u001b[39m'\u001b[39m\u001b[39mproject_list\u001b[39m\u001b[39m'\u001b[39m) \u001b[39mas\u001b[39;00m f:\n\u001b[0;32m      4\u001b[0m     \u001b[39mfor\u001b[39;00m project \u001b[39min\u001b[39;00m f\u001b[39m.\u001b[39mread()\u001b[39m.\u001b[39msplit(\u001b[39m'\u001b[39m\u001b[39m\\n\u001b[39;00m\u001b[39m'\u001b[39m):\n\u001b[1;32m----> 5\u001b[0m         scrap \u001b[39m=\u001b[39m scrapping\u001b[39m.\u001b[39;49mget_raw_english_texts_of_project(project)\n\u001b[0;32m      6\u001b[0m         \u001b[39m# connection = connection.execute(\"insert into raw_texts values (%(id_text)s, %(project_name)s, %(raw_text)s)\", scrap)\u001b[39;00m\n\u001b[0;32m      7\u001b[0m         \u001b[39mtry\u001b[39;00m:\n",
+      "File \u001b[1;32mc:\\Users\\Saret\\Programming\\C#\\DH\\scrapping.py:43\u001b[0m, in \u001b[0;36mget_raw_english_texts_of_project\u001b[1;34m(project_dirname)\u001b[0m\n\u001b[0;32m     41\u001b[0m \u001b[39mfor\u001b[39;00m filename \u001b[39min\u001b[39;00m all_paths:\n\u001b[0;32m     42\u001b[0m     cur_json \u001b[39m=\u001b[39m _load_json_from_path(filename)\n\u001b[1;32m---> 43\u001b[0m     project_name \u001b[39m=\u001b[39m cur_json[\u001b[39m'\u001b[39;49m\u001b[39mproject\u001b[39;49m\u001b[39m'\u001b[39;49m]\n\u001b[0;32m     45\u001b[0m     \u001b[39m# # Skip in case we are in saa project and the current sub project is not in neo-assyrian\u001b[39;00m\n\u001b[0;32m     46\u001b[0m     \u001b[39m# if project_dirname == \"saao\" and project_name[-2:] not in SUB_PROJECTS_IN_NEO_ASS:  # TODO: validate\u001b[39;00m\n\u001b[0;32m     47\u001b[0m     \u001b[39m#     continue\u001b[39;00m\n\u001b[0;32m     48\u001b[0m \n\u001b[0;32m     49\u001b[0m     \u001b[39m# id_text = member.get('id_text', \"\") + member.get('id_composite', \"\")\u001b[39;00m\n\u001b[0;32m     50\u001b[0m     \u001b[39m# html_dir = \"/\".join(path.parts[1:-1])\u001b[39;00m\n\u001b[0;32m     51\u001b[0m     url \u001b[39m=\u001b[39m \u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mhttp://oracc.iaas.upenn.edu/\u001b[39m\u001b[39m{\u001b[39;00mproject_name\u001b[39m}\u001b[39;00m\u001b[39m/\u001b[39m\u001b[39m{\u001b[39;00mcur_json[\u001b[39m'\u001b[39m\u001b[39mtextid\u001b[39m\u001b[39m'\u001b[39m]\u001b[39m}\u001b[39;00m\u001b[39m/html\u001b[39m\u001b[39m\"\u001b[39m\n",
+      "\u001b[1;31mTypeError\u001b[0m: 'NoneType' object is not subscriptable"
+     ]
+    }
+   ],
    "source": [
     "conn = psycopg2.connect(\"dbname='dh' user='dh' host='dh.saret.tk' password='qwerty'\")\n",
     "connection = conn.cursor()\n",
@@ -48,6 +61,41 @@
     "            logging.error(f\"Error in {project}:{Exception}\")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "conn = psycopg2.connect(\"dbname='dh' user='dh' host='dh.saret.tk' password='qwerty'\")\n",
+    "connection = conn.cursor()\n",
+    "logging.basicConfig(\n",
+    "    level=logging.INFO, filename=f'./now.log',\n",
+    "    format='%(name)s - %(levelname)s - %(message)s')\n",
+    "logging.info('Start')\n",
+    "with open('project_list') as f:\n",
+    "    project = f.read().split('\\n')[2]\n",
+    "    scrap = scrapping.get_raw_english_texts_of_project(project)\n",
+    "    execute_batch(connection, \"insert into raw_texts values (%(id_text)s, %(project_name)s, %(raw_text)s)\", scrap)\n",
+    "    conn.commit()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
   {
    "cell_type": "code",
    "execution_count": null,

scrapping.py

@@ -9,7 +9,10 @@ import glob
 import logging
 # parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 # os.chdir(parent_dir)
-logging.basicConfig(filename='scrape.log', level=logging.INFO)
+import datetime
+logging.basicConfig(
+    level=logging.INFO, filename=f'{datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}.log', filemode='w',
+    format='%(name)s - %(levelname)s - %(message)s', datefmt='%d-%b-%y %H:%M:%S')
 JSONS_DIR = "jsons_unzipped"
 
 SUB_PROJECTS_IN_NEO_ASS = ["01", "05", "06", "07", "09", "11", "12", "14", "15", "16"]