Email CSV Data download and Processing to Google Sheets

I gang Opslået 4 år siden Betalt ved levering
I gang Betalt ved levering

These importing and processing tasks should happen automatically, once daily.

-Interface with gmail to obtain specific emails that have arrived since last check

-These can be coordinated with me to have them receive a specific label, or go into a folder, or other special filtering methods to make it easier to process.

Download from 1 to 5 raw CSV files from the Email links.

• There will only be links for the files that are available

• Example email raw file is provided

Import data:

• Put the raw data, from each of the 4 files into some sort of database for future reference

• For your information: The Packet file is extremely large - usually around 16 megabytes of TEXT, containing up to 100,000 rows per date included in the file (up to 7 days will be in each file)

• There will be DATES/TIME overlap of the data rows in each file. Don’t store the data for rows that have already been imported.

◦ The file contains data that begins collecting at the beginning of the current week, and all data throughout the week is simply appended to the file.

◦ For example, if the file is downloaded on Monday, it may contain only the data from Monday. When downloaded again on Tuesday, it may contain Monday as well as Tuesday data.

◦ The rows in the files will always be in chronological order, with new data at the bottom of the file.

Process the data:

Put the output data into some sort of database for future reference:

For all files:

• Change time zone from UTC to US/Los Angeles time zone (automatically adjust for Daylight Savings Time in US/Los Angeles, and be careful - there is no daylight savings time in UTC)

• Retain the same date/time text formatting as the import file

Packet File:

• Reduce the number of data columns to: Date, Zone, HRV, BPM

• Reduce the number of data rows by AVERAGING column values in 1 minute time intervals, starting with second :00 and ending with second :59 of each clock integer minute.

◦ For example: Output a data line time stamped 19:43:00 containing the average of all samples having time stamps between 19:42:00.0 to 19:42:59.9

Interventions File:

• No additional processing

Snaps file:

• No additional processing

Assessments file:

• No additional processing

Thought Records file:

• No additional processing

Retrieve data on demand:

Data output format:

• Direct populate a google sheet/cells

◦ Sheet that function is called from, or via some other web API site such as Apify, zapier, integromat etc that can access google sheets.

• CSV text

• JSON

User Interface:

Method for triggering the retrieval should be discussed, and agreed upon before implementation- suggest a method or two.

Adjusting settings at this phase of the project should be limited to simple OPTIONAL variable inputs loaded at runtime:

• Possible examples:

◦ passing arguments to a function in a function call: functionname(20190321,12,5)

◦ Reading from a preferences file

◦ Reading from a set of google sheet cells

Input variables. Variables should be optional, and if not provided, default to the value provided here:

• Starting and ending date/time for retrieved processed data

◦ Format: (same as imported date/time format)

▪ YYYY-MM-dd HH:mm:ss

▪ Example: 2019-03-17 00:17:32

◦ Default to: 0:00 to 23:59 of the current date in the US/Los Angeles time zone. (Not the current date GMT)

• Data record type

◦ Options: Packet, Interventions, Snaps, Assessments

◦ Format: p, i, s, or a character

◦ Default to: Packet (p)

• Error reporting format

◦ Human readable or machine readable

◦ Format: h, m character

◦ Default to: human (h)

Error handling:

• Unparsable date/time range

• Variable input not recognized

• Other errors as needed

• Use your best judgement on the format of the machine readable error messages.

Google App Engine

Projekt ID: #19036022

Om projektet

1 bud Remote projekt Aktiv 4 år siden

Tildelt til:

mrbrave

Hi, As discussed, we will do 2 milestones. First for csv processing with completion time 3 days. Second for filetring and output with completion time 3 days. Further testing and making process 100% perfect may Flere

$330 USD in 10 dage
(7 bedømmelser)
4.4