==========================================
* ``prototype`` : this directory contains scripts to perform an experimental
- import of the CREM database into Telemeta 0.3.x. It is only meant as a
+ import of the CREM/MUCEM database into Telemeta 0.3.x. It is only meant as a
proof of concept.
* ``raw_conversion`` : instructions and scripts to perform a full raw conversion
of the CREM database from 4D to MySQL. The result has the same structure
+++ /dev/null
-==================================================
-CREM data and how to import it into Telemeta 0.3.x
-==================================================
-
-Warning: the following instructions allow you to import the CREM's data
-into telemeta. However, it is currently a rather data destructive process,
-only meant for demonstration and testing purpose. Do not use this in a
-production environment.
-
-1. Install Telemeta upon MySQL. Initialize the database using Telemeta's
- Django models. Ensure that everything is running fine before going any
- further.
-
-2. Run prepare on the source directory. The source directory must contain
- text files as they were exported from the 4D database. These files are
- expected to be gzipped.
-
- Example::
-
- $ scripts/prepare src/2007-05-09
-
-3. Use the import.sql script to insert the data into your MySQL database.
-
- Example::
-
- $ mysql your_telemeta_database < scripts/import.sql
-
-4. Copy the WAV test file of your choice, into <MEDIA_ROOT>/items/test.wav
- This single file is associated with all media items, for testing purpose.
-
-That should be it. If you want to run Telemeta against SQLite instead of
-MySQL, first follow the above instructions to import the data into MySQL.
-Then convert your data from MySQL to SQLite, this is a common task, google
-about it.
-
-
-
--- /dev/null
+==================================================
+CREM data and how to import it into Telemeta 0.3.x
+==================================================
+
+Warning: the following instructions allow you to import the CREM's data
+into telemeta. However, it is currently a rather data destructive process,
+only meant for demonstration and testing purpose. Do not use this in a
+production environment.
+
+1. Install Telemeta upon MySQL. Initialize the database using Telemeta's
+ Django models. Ensure that everything is running fine before going any
+ further.
+
+2. Run prepare on the source directory. The source directory must contain
+ text files as they were exported from the 4D database. These files are
+ expected to be gzipped.
+
+ Example::
+
+ $ scripts/crem-prepare src/crem-2007-05-09
+
+3. Use the SQL import script to insert the data into your MySQL database.
+
+ Example::
+
+ $ mysql your_telemeta_database < scripts/crem-import.sql
+
+4. Copy the WAV test file of your choice, into <MEDIA_ROOT>/items/test.wav
+ This single file is associated with all media items, for testing purpose.
+
+That should be it. If you want to run Telemeta against SQLite instead of
+MySQL, first follow the above instructions to import the data into MySQL.
+Then convert your data from MySQL to SQLite, this is a common task, google
+about it.
+
+
+
--- /dev/null
+==================================================
+MUCEM data and how to import it into Telemeta 0.3.x
+==================================================
+
+Warning: the following instructions allow you to import the MUCEM's data
+into telemeta. However, it is currently a rather data destructive process,
+only meant for demonstration and testing purpose. Do not use this in a
+production environment.
+
+1. Install Telemeta upon MySQL. Initialize the database using Telemeta's
+ Django models. Ensure that everything is running fine before going any
+ further.
+
+2. Copy the csv data into /tmp/mucem_import.csv
+
+ Example::
+
+ $ cp src/mucem-2009-15-01/ItemCollMucem15-01.csv /tmp/mucem_import.csv
+
+3. Use the SQL import script to insert the data into your MySQL database.
+
+ Example::
+
+ $ mysql your_telemeta_database < scripts/mucem-import.sql
+
+4. Copy the WAV test file of your choice, into <MEDIA_ROOT>/items/test.wav
+ This single file is associated with all media items, for testing purpose.
+
+That should be it. If you want to run Telemeta against SQLite instead of
+MySQL, first follow the above instructions to import the data into MySQL.
+Then convert your data from MySQL to SQLite, this is a common task, google
+about it.
+
+
+
--- /dev/null
+DELETE FROM telemeta_collection;
+LOAD DATA INFILE '/tmp/crem_import/support.txt' INTO TABLE telemeta_collection
+FIELDS TERMINATED BY '\t'
+LINES TERMINATED BY '\r\n';
+
+UPDATE telemeta_collection SET id=REPLACE(id, '/', '--');
+UPDATE telemeta_collection SET id=REPLACE(id, '>', 'gt');
+UPDATE telemeta_collection SET id=REPLACE(id, ' ', '');
+UPDATE telemeta_collection SET id=REPLACE(id, '*', '');
+UPDATE telemeta_collection SET copied_from_id = NULL WHERE copied_from_id = '';
+
+DELETE FROM telemeta_item;
+LOAD DATA INFILE '/tmp/crem_import/phono.txt' INTO TABLE telemeta_item
+FIELDS TERMINATED BY '\t'
+LINES TERMINATED BY '\r\n';
+
+UPDATE telemeta_item SET id=REPLACE(id, ":", "__");
+UPDATE telemeta_item SET id=REPLACE(id, '/', '--');
+UPDATE telemeta_item SET id=REPLACE(id, '?', 'i');
+UPDATE telemeta_item SET id=REPLACE(id, '>', 'gt');
+UPDATE telemeta_item SET id=REPLACE(id, ' ', '_');
+UPDATE telemeta_item SET id=REPLACE(id, '*', '');
+UPDATE telemeta_item SET id=REPLACE(id, '(', '');
+UPDATE telemeta_item SET id=REPLACE(id, ')', '');
+UPDATE telemeta_item SET id=REPLACE(id, '&', '');
+UPDATE telemeta_item SET id=REPLACE(id, ',', '.');
+UPDATE telemeta_item SET id=REPLACE(id, '!', '');
+UPDATE telemeta_item SET collection_id=REPLACE(collection_id, '/', '--');
+UPDATE telemeta_item SET title='' WHERE title='N';
+UPDATE telemeta_item SET etat=REPLACE(etat, ')', '_');
+UPDATE telemeta_item SET etat=REPLACE(etat, '(', '_');
+UPDATE telemeta_item SET file="items/test.wav";
+
+DELETE FROM telemeta_physicalformat;
+INSERT INTO telemeta_physicalformat (value)
+ SELECT DISTINCT physical_format FROM telemeta_collection
+ WHERE physical_format <> '' AND physical_format IS NOT NULL;
+
+DELETE FROM telemeta_publishingstatus;
+INSERT INTO telemeta_publishingstatus (value)
+ SELECT DISTINCT publishing_status FROM telemeta_collection
+ WHERE publishing_status <> '' AND publishing_status IS NOT NULL;
--- /dev/null
+#!/bin/bash
+
+if [ "$1" == "" ]
+then
+ echo "Please provide the source directory"
+ exit 1
+fi
+
+src=$1
+tmpdir=crem_import
+required="support.txt.gz phono.txt.gz"
+
+for f in $required
+do
+ if ! [ -f $src/$f ]
+ then
+ echo "Can't find $f in $src"
+ exit 1
+ fi
+done
+
+if [ -d /tmp/$tmpdir ]
+then
+ rm /tmp/$tmpdir/*
+else
+ mkdir /tmp/$tmpdir
+fi
+
+echo -n "Converting charset and cleaning text files.. "
+
+zcat $src/support.txt.gz | tail -n +2 \
+ | sed 's/^ *//' | sed 's/ *\t */\t/g'| sed 's/ *$//' \
+ | iconv -f WINDOWS-1252 -c -t latin1 \
+ > /tmp/$tmpdir/support.txt
+
+zcat $src/phono.txt.gz | tail -n +2 \
+ | sed 's/^ *//' | sed 's/ *\t */\t/g'| sed 's/ *$//' \
+ | iconv -f WINDOWS-1252 -c -t latin1 | sed '/^\t/d' \
+ > /tmp/$tmpdir/phono.txt
+
+echo "Done"
+
+
+++ /dev/null
-DELETE FROM telemeta_collection;
-LOAD DATA INFILE '/tmp/crem_import/support.txt' INTO TABLE telemeta_collection
-FIELDS TERMINATED BY '\t'
-LINES TERMINATED BY '\r\n';
-
-UPDATE telemeta_collection SET id=REPLACE(id, '/', '--');
-UPDATE telemeta_collection SET id=REPLACE(id, '>', 'gt');
-UPDATE telemeta_collection SET id=REPLACE(id, ' ', '');
-UPDATE telemeta_collection SET id=REPLACE(id, '*', '');
-UPDATE telemeta_collection SET copied_from_id = NULL WHERE copied_from_id = '';
-
-DELETE FROM telemeta_item;
-LOAD DATA INFILE '/tmp/crem_import/phono.txt' INTO TABLE telemeta_item
-FIELDS TERMINATED BY '\t'
-LINES TERMINATED BY '\r\n';
-
-UPDATE telemeta_item SET id=REPLACE(id, ":", "__");
-UPDATE telemeta_item SET id=REPLACE(id, '/', '--');
-UPDATE telemeta_item SET id=REPLACE(id, '?', 'i');
-UPDATE telemeta_item SET id=REPLACE(id, '>', 'gt');
-UPDATE telemeta_item SET id=REPLACE(id, ' ', '_');
-UPDATE telemeta_item SET id=REPLACE(id, '*', '');
-UPDATE telemeta_item SET id=REPLACE(id, '(', '');
-UPDATE telemeta_item SET id=REPLACE(id, ')', '');
-UPDATE telemeta_item SET id=REPLACE(id, '&', '');
-UPDATE telemeta_item SET id=REPLACE(id, ',', '.');
-UPDATE telemeta_item SET id=REPLACE(id, '!', '');
-UPDATE telemeta_item SET collection_id=REPLACE(collection_id, '/', '--');
-UPDATE telemeta_item SET title='' WHERE title='N';
-UPDATE telemeta_item SET etat=REPLACE(etat, ')', '_');
-UPDATE telemeta_item SET etat=REPLACE(etat, '(', '_');
-UPDATE telemeta_item SET file="items/test.wav";
-
-DELETE FROM telemeta_physicalformat;
-INSERT INTO telemeta_physicalformat (value)
- SELECT DISTINCT physical_format FROM telemeta_collection
- WHERE physical_format <> '' AND physical_format IS NOT NULL;
-
-DELETE FROM telemeta_publishingstatus;
-INSERT INTO telemeta_publishingstatus (value)
- SELECT DISTINCT publishing_status FROM telemeta_collection
- WHERE publishing_status <> '' AND publishing_status IS NOT NULL;
--- /dev/null
+
+SET NAMES 'utf8';
+
+DROP TABLE IF EXISTS mucem_import;
+CREATE TABLE mucem_import (
+ f1 VARCHAR (255) NOT NULL, f2 VARCHAR (255) NOT NULL, f3 VARCHAR (255) NOT NULL,
+ f4 VARCHAR (255) NOT NULL, f5 VARCHAR (255) NOT NULL, f6 VARCHAR (255) NOT NULL,
+ f7 VARCHAR (255) NOT NULL, f8 VARCHAR (255) NOT NULL, f9 VARCHAR (255) NOT NULL,
+ f10 VARCHAR (255) NOT NULL, f11 VARCHAR (255) NOT NULL, f12 VARCHAR (255) NOT NULL,
+ f13 VARCHAR (255) NOT NULL, f14 VARCHAR (255) NOT NULL, f15 VARCHAR (255) NOT NULL,
+ f16 VARCHAR (255) NOT NULL, f17 VARCHAR (255) NOT NULL, f18 VARCHAR (255) NOT NULL,
+ f19 VARCHAR (255) NOT NULL, f20 VARCHAR (255) NOT NULL, f21 VARCHAR (255) NOT NULL,
+ f22 VARCHAR (255) NOT NULL, f23 VARCHAR (255) NOT NULL, f24 VARCHAR (255) NOT NULL,
+ f25 VARCHAR (255) NOT NULL, f26 VARCHAR (255) NOT NULL, f27 VARCHAR (255) NOT NULL,
+ f28 VARCHAR (255) NOT NULL, f29 VARCHAR (255) NOT NULL, f30 VARCHAR (255) NOT NULL,
+ f31 VARCHAR (255) NOT NULL, f32 VARCHAR (255) NOT NULL, f33 VARCHAR (255) NOT NULL,
+ f34 VARCHAR (255) NOT NULL, f35 VARCHAR (255) NOT NULL, f36 VARCHAR (255) NOT NULL,
+ f37 VARCHAR (255) NOT NULL, f38 VARCHAR (255) NOT NULL, f39 VARCHAR (255) NOT NULL,
+ f40 VARCHAR (255) NOT NULL, f41 VARCHAR (255) NOT NULL, f42 VARCHAR (255) NOT NULL,
+ f43 VARCHAR (255) NOT NULL, f44 VARCHAR (255) NOT NULL, f45 VARCHAR (255) NOT NULL,
+ f46 VARCHAR (255) NOT NULL, f47 VARCHAR (255) NOT NULL, f48 VARCHAR (255) NOT NULL,
+ f49 VARCHAR (255) NOT NULL, f50 VARCHAR (255) NOT NULL, f51 VARCHAR (255) NOT NULL
+);
+
+LOAD DATA INFILE '/tmp/mucem_import.csv' INTO TABLE mucem_import
+ CHARACTER SET 'utf8' FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '"';
+
+DELETE FROM telemeta_item;
+INSERT INTO telemeta_item
+SELECT f1, f2, f3, f4, f5, f6, f7, f8, CONCAT_WS(' - ', f46, f9, f43), f10,
+ f11, f12, f13, f14, f16, f17, f18, f19, f20, f22, f23, f24, f25, f26, f27,
+ f28, f29, f30, "", f31, 'items/test.wav' FROM mucem_import;
+UPDATE telemeta_item SET continent = UPPER(continent);
+
+DELETE FROM telemeta_collection;
+INSERT INTO telemeta_collection (physical_format, id, title, physical_items_num,
+ creator, booklet_writer)
+SELECT f2, f3, f11, "1", f25, f47 FROM mucem_import;
+
+DELETE FROM telemeta_physicalformat;
+INSERT INTO telemeta_physicalformat (value)
+ SELECT DISTINCT physical_format FROM telemeta_collection
+ WHERE physical_format <> '' AND physical_format IS NOT NULL;
+
+++ /dev/null
-#!/bin/bash
-
-if [ "$1" == "" ]
-then
- echo "Please provide the source directory"
- exit 1
-fi
-
-src=$1
-tmpdir=crem_import
-required="support.txt.gz phono.txt.gz"
-
-for f in $required
-do
- if ! [ -f $src/$f ]
- then
- echo "Can't find $f in $src"
- exit 1
- fi
-done
-
-if [ -d /tmp/$tmpdir ]
-then
- rm /tmp/$tmpdir/*
-else
- mkdir /tmp/$tmpdir
-fi
-
-echo -n "Converting charset and cleaning text files.. "
-
-zcat $src/support.txt.gz | tail -n +2 \
- | sed 's/^ *//' | sed 's/ *\t */\t/g'| sed 's/ *$//' \
- | iconv -f WINDOWS-1252 -c -t latin1 \
- > /tmp/$tmpdir/support.txt
-
-zcat $src/phono.txt.gz | tail -n +2 \
- | sed 's/^ *//' | sed 's/ *\t */\t/g'| sed 's/ *$//' \
- | iconv -f WINDOWS-1252 -c -t latin1 | sed '/^\t/d' \
- > /tmp/$tmpdir/phono.txt
-
-echo "Done"
-
-
--- /dev/null
+,"Disque78T, Ø25 cm ","47.6",1,"MNATP_1947_06_01",,"8/09/1947","France","Basses-Pyrénées","Basques","mission Pays Basque, 1947","Improvisation : salutation
+1 voix d'homme",,,"1 voix d'homme",,,,,,,,,,"Claudie Marcel-Dubois et Maguy Andral (mission Pays Basque, 1947)"," Pays Basque",,,,,"Europe",,,,"Disque","Mono",,,,,"78 rpm","25 cm Ø","Pyrénées Atlantiques","Non précisé","Phonothèque du MuCEM","Bayonne - Saint-Jean-Pied-de-Port","Raphaël Parejo Coudert"
+,"Disque78T, Ø25 cm ","52.22",1,"MNATP_1952_22_01",,"1947","France","Basses-Pyrénées","Basques","mission Pays Basque, 1947","Ene matiak igorri (Mon aimée m'a envoyé)
+1 voix d'homme",,,,,,,,,,,,,"Claudie Marcel-Dubois et Maguy Andral (mission Pays Basque, 1947)"," Pays Basque",,,,,"Europe",,,,"Disque","Mono",,,,,"78 rpm","25 cm Ø","Pyrénées Atlantiques","Irigaray (B.)","Phonothèque du MuCEM","St Jean-Pied-de-Port","Raphaël Parejo Coudert"
+,"Bobine de 17 cm de diam.","53.8",1,"MNATP_1953_08_01",,"12/07/1953","France","Basses-Pyrénées","Basques","mission Pays Basque, 1952","Saut Basque
+Clarinette, 2 trompettes.",,,,,,,,,,,,,"Melles Cl. Marcel-Dubois et M. Andral"," Pays Basque",,,,,"Europe",,,,"Bande magnétique","Mono",,,,,"19 cm/s","Bobine de 17 cm de diam.","Pyrénées Atlantiques","Groupe de danseurs Bas-Navarrais de Hélette","Phonothèque du MuCEM","Bayonne - Iholdy - Helette","Raphaël Parejo Coudert"
+,"Bobine de 17 cm de diam.","53.9",1,"MNATP_1953_09_O1",,"12/07/1953","France","Basses-Pyrénées","Basques","mission Pays Basque, 1953","Braletik
+Chirula et tuntun.",,,,,,,,,,,,,"Melles Cl. Marcel-Dubois et M. Andral"," Pays Basque",,,,,"Europe",,,,"Bande magnétique","Mono",,,,,"20 cm/s","Bobine de 17 cm de diam.","Pyrénées Atlantiques","Patalagoïty (Simon)",,"Bayonne - Oloron - Tardets - Lacarre","Raphaël Parejo Coudert"
+,"Bobine de 17 cm de diam.","54.4",1,"MNATP_1954_04_01",,"1953","France","Basses-Pyrénées","Basques"," J. Séguy et J. Allières","Bûche de Noël
+1 voix d'homme parlée.",,,,,,,,,,,,,"Professeurs J. Séguy et J. Allières"," Pays Basque",,,,,"Europe",,,,"Bande magnétique","Mono",,,,,"19 cm/s","Bobine de 17 cm de diam.","Pyrénées Atlantiques","Candelé (Étienne)","Phonothèque du MuCEM","Bayonne - Labastide-Clairence","Raphaël Parejo Coudert"
+,"Bobine de 12 cm de diam.","58.12",1,"MNATP_1958_12_01",,"03/10/1957","France","Basses-Pyrénées","Basques","Musée Basque de Bayonne","Cantique pendant la messe
+Ensemble de voix d'hommes.",,,,,,,,,,,,,"? Don du Musée Basque de Bayonne"," Pays Basque",,,,,"Europe",,,,"Bande magnétique","Mono",,,,,"19 cm/s","Bobine de 12 cm de diam.","Pyrénées Atlantiques","Chorale Gaztelu Zahar","Phonothèque du MuCEM","Saint-Étienne-de-Baïgorry","Raphaël Parejo Coudert"
+,"Bobine de 12 cm de diam.","60.3",1,"MNATP_1960_03_01",,"1959","France","Basses-Pyrénées","Basques","Duhourcau (B.) et Guichard (Cl.)","Sonnailles en Pays Basque : commentaire d'introduction
+1 voix d'homme parlée, cloches et sonnailles.",,,,,,,,,,,,,"Duhourcau (B.) et Guichard (Cl.)"," Pays Basque",,,,,"Europe",,,,"Bande magnétique","Mono",,,,,"19 cm/s","Bobine de 12 cm de diam.","Pyrénées Atlantiques","Duhourcau (B.)","Phonothèque du MuCEM","Saint-Jean-le-Vieux","Raphaël Parejo Coudert"
+,"Bobine de 17 cm de diam.","64.35",1,"MNATP_1964_35_01",,"24/05/1964","France","Basses-Pyrénées","Basques","J. M. Guilcher, Mission ATP - CNRS, 1964","Dantza korola
+Voix sifflée.",,,,,,,,,,,,,"J. M. Guilcher (Mission ATP - CNRS)"," Pays Basque",,,,,"Europe",,,,"Bande magnétique",,,,,,"19 cm/s","Bobine de 17 cm de diam.","Pyrénées Atlantiques","Bertzaix (Martin)","Phonothèque du MuCEM","Bayonne - Behorleguy","Raphaël Parejo Coudert"
+,"Bobine de 17 cm de Ø","68.58",1,"MNATP_1968_58_01",,"03/03/1968","France","Pays Basque","Basques","Guilcher J.-M., Lajoux J.-D. - Mission Basses-Pyrénées","Mascarade souletine : jeux de l'après-midi
+Flûte, tambour, clochettes",,,,,,,,,,,,,"Guilcher J.-M., Lajoux J.-D. - Mission Basses-Pyrénées"," Pays Basque",,,,,"Europe",,,,"Bande magnétique",,,,,,"9,5 cm/s","Bobine de 17 cm de Ø","Pyrénées Atlantiques",,"Phonothèque du MuCEM","Pagolle","Raphaël Parejo Coudert"
+,"Bobine de 18 cm de Ø","72.45",1,"MNATP_1972_45_01",,"03/03/1968","France","Pays Basque","Basques","Guilcher J.-M., Lajoux J.-D. - Mission Basses-Pyrénées","Mascarade souletine : arrivée de la mascarade.Voix parlées, ambiance, cloche, orgue, choeur mixte",,,,,,,,,,,,,"Lajoux J.-D. - Tournage d'un film sur la mascarade souletine"," Pays Basque",,,,,"Europe",,,,"Bande magnétique",,,,,,"19 cm/s","Bobine de 18 cm de Ø","Pyrénées Atlantiques",,"Phonothèque du MuCEM","Pagolle","Raphaël Parejo Coudert"
+,"Bobine de 18 cm de Ø","73.35",1,"MNATP_1973_35_01",,28 July 1973,"France","Basses-Pyrénées","Basques","Claudie Marcel-Dubois et Marguerite Andral - Mission CNRS-ATP","Ingurutxo de Luzia, enchaîné à la Danse des mouchoirs
+2 txistus, 1 tamboril, 1 tambour plat",,,,,,,,,,,,,"Claudie Marcel-Dubois et Marguerite Andral - Mission CNRS-ATP"," Pays Basque",,,,,"Europe",,,,"Bande magnétique",,,,,,"19 cm/s","18 cm de diamètre","Pyrénées Atlantiques","Groupe de Vitoria Gazttetxo","Phonothèque du MuCEM","Bayonne, fronton ""Romería""","Raphaël Parejo Coudert"
projects / telemeta-data.git / commitdiff