DataUpload
APIs for collecting or uploading data. There are two main APIs, one for uploading
Large files (bulk, millions of records) and another for sending real-time, or slightly batched
events (Less than 10MB).
**stream**
A stream name is like a table in database, each stream is a differnt _Type_ of data.
You assign a name to your data uploads to identify each stream/type.
However, it should be consistent across uploads of same type of data.
**Timestamp Fields**
When uploading historical, or event data you may have a time/date at which
an event occured. Normal event uploads get an event timestamp at the server time
but you can over-ride this behavior so the timestamp is defined in the event you upload.
This is a _timestamp_field_ parameter in api's below. The formatting for this is very, very
lenient with over 50 format's supported.
BulkCSVUpload
https://api.lytics.io/collect/bulk/{stream}Upload CSV files (may be gzipped). They MUST contain headers: either
in first row (by default) OR in **headers** querystring field.
If the file contains a timestamp field you want us to use
see **timestamp_field** querystring parameter for how we extract it.
*Note Domain* this api is on a different domain *https://bulk.lytics.io* because the *api.lytics.io* domain
is limited to 10MB file uploads.
``sh
# the "user_regdata" is the STREAM (ie Type) of data, if there
# is multiple different types of data (userdata, emaildata, events)
# they would have different streams.
# user data
export LIODATAKEY="YOURKEY"
curl -s -H "Authorization: $LIOKEY" \
-H 'Content-type: application/csv' \
--data-binary @users.csv \
"https://bulk.lytics.io/collect/bulk/user_regdata?timestamp_field=register_date" \
| jq '.'
# only do a dry-run to see if it is formatted correctly
curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?access_token=$LIODATAKEY×tamp_field=register_date&dryrun=true" \
--data-binary @users.csv -H 'Content-type: application/csv' | jq '.'
# append fields to each row
# where append = urlencode({"source":"crm"})
# often used when file-name contains information
curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?append=%7B%22source%22%3A%22crm%22%7D&access_token=$LIODATAKEY" \
--data-binary @users.csv -H 'Content-type: application/csv' | jq '.'
# full example uploading gzipped file
echo 'user_id,email,reg_date,item_count
7456,[email protected],"2012-10-17T17:29:39.738Z",82
1234,[email protected],"2012-12-11T19:53:31.547Z",82' > users.csv
gzip users.csv
curl -s "https://bulk.lytics.io/collect/bulk/user_regdata?access_token=$LIODATAKEY×tamp_field=reg_date&dryrun=true&filename=users.csv" \
--data-binary @users.csv.gz \
-H 'Content-type: application/csv' | jq '.'
``
{
"status": 200,
"message": "success",
"data": {
"message_count": 2,
"rejected_count": 0,
"content-type": "application/json",
"droperrors": false,
"dryrun": false,
"timestamp_field": "event_created",
"filename": "myfile.csv"
}
}BulkJSONUpload
https://api.lytics.io/collect/bulk/{streamname}JSON formatting info.
This JSON is ok:
``json
{"order_ids":[1,2,3]}
`
This is not valid because it has objects nested inside arrays:
`json
{"orders":[{"id":1,"price":22},{"id":2,"price":23}]}
``
{
"status": 200,
"message": "success",
"data": {
"message_count": 2,
"rejected_count": 0,
"content-type": "application/json",
"droperrors": false,
"dryrun": false,
"timestamp_field": "event_created",
"filename": "myfile.json"
}
}Data JSON Upload
https://api.lytics.io/collect/json/{stream}{
"status": 200,
"message": "success",
"data": {
"message_count": 2,
"rejected_count": 0,
"content-type": "application/json",
"droperrors": false,
"dryrun": false,
"timestamp_field": "event_created",
"filename": "myfile.json"
}
}