Influxdb Tutoiral Series - Python API Client with Pandas to export CSV data into Influxdb

How to export CSV data into influxdb by using Influxdb API client. Python library pandas is used along with influxdb python API client to do export......

Topic covered -
1. Pandas for manipulating CSV file
2. Conversion of CSV records into list of JSON format that is required to process by InfluxDB client
3. Export CSV records into InfluxDB
4. Verification of exported data into InfluxDB client

[havalak]

Hi Abishek,
You mentioned in your video lastly there would be another video using a larger excel file. I could not find it anywhere. Can you please help me?

[Playstationize]

How can I export influx data to a csv file? I tried the query select: SELECT * FROM "db_name" WHERE time> '2017-09-05T23: 59: 59Z' and time <'2017-09-15T23: 59: 59Z' -format csv> data .csv

but it does not work

[SoumilShah]

Hello, i have a csv file with 3 lac rows this methods fails to load the dataset after 30, 000 rows any suggestion ?

[anshulchoudhary]

@abhishek - I need to upload around 2 million rows which is a very time-consuming process, can you please guide?

[abhishekthetechexplorer]

Hi Swetha, Issue1 Resolution : should be inside the for loop as currently it is happening outside for loop that is the reason you are getting only last row.... Issue2 Resolution : dont use row[2] because it means you are picking up 3rd column make it more generic logic based on what all fields you want to extract & store in db... i hope this will help you

[abhishekthetechexplorer]

There can be many ways to do so however could you please be more specific in terms of frequency like daily this upload would happen or data keep coming regularly or storing on some file server from there it needs to picked & stored... Provide more detail then I can help you to improve on time taken for upload data into influxDB

[aloho-mora]

Hi Abhishek, this is very informative. As I read from the comments you were going to include the downloadable available on github. Can i please know your git handle?

[swethagorantla]

Hi Abhishek - By following the video, i could import data into InfluxDB successfully. But i am facing 2 issues as below
1) For loop is converting only last row of csv data into Json
2) In my CSV many columns are there but script is taking only value in column3.

I know the change required in for loop, Tried many ways but didnt succeeded can you please help me in resolving this issue?

My script is below -

import pandas as pd
from influxdb import InfluxDBClient

client = InfluxDBClient(host='localhost', port=8086)


file_path =

csvReader = pd.read_csv(file_path)

#print(csvreader.shape)
#print(csvreader.columns)

for row_index, row in csvReader.iterrows():
tags = row[0]
fieldvalue = row[2]
json_body = [
{
"measurement": "LEGO_throughput",
"tags": {
"Reference": tags
},
"fields": {
"value": fieldvalue
}
}
]




and my CSV data -

series(eventTimestamp), count(*), "percentile(responseTime, 95)", avg(responseTime)
2020-07-17T01:17:00+01:00, 81, 739, 444.9753086
2020-07-17T01:18:00+01:00, 784, 2600, 809.3762755
2020-07-17T01:19:00+01:00, 3127, 2825, 1316.033259
2020-07-17T01:20:00+01:00, 6348, 2908, 1421.663674

[hotnewsforyou]

can you explain how to save web-scrapping dataframe into influxdb

[abhishekthetechexplorer]

Why not...will definitely do that....but for this weekend I already have plan but in future would do that....

[faizrsl]

Hi Sir, can you cover on how to do a more complex query on influxdb such as "where time = 'XX:XX' ", "into Y", "group by Z" using python and store the data retrieved using pandas? Love your vid. Keep up the good work :D

[saifkhan-jkxf]

I have a file with 1100 column how do I insert this into Influxdb.... need help if anyone can

[swethagorantla]

HI Abhishek - Thanks for the session. Its very clear and informative. One of the prerequisite is "InfluxDB server should run on 8085 by default". Can i know how to check this precondition. FYI, I am using AWS windows server 2019 data center machine to do this task.

[AlexGarbino]

Can you make your example code downloadable, eg in a github repo?

[anshulchoudhary]

hi Abhisekh, thanks that was really nice session. Why havent you added the time in the JSON body. could have made all the things really easy. It diffcult always to handle timestamp data :)