Search for a command to run...
Articles tagged with #python
Spark SQL is a component of Apache Spark that works with tabular data. # Load data from file df = spark.read.csv("trains.csv", header=True) # Create temporary table called table1 df.createOrReplaceTempView("trains") # Inspect the columns in the tab...
To interact with AWS in Python, there is the Boto3 library. 1. AWS S3 S3 is the AWS Storage solution. import boto3 # Generate the boto3 client for interacting with S3 s3 = boto3.client('s3', region_name='eu-south-1', #region where your reso...
Spark = tool for doing parallel computation with large datasets. Spark lets you spread data and computations over clusters with multiple nodes.pyspark = Python package that integrate Spark with Python. The SparkSession.builder.getOrCreate() method re...
Airflow: platform to program workflows. DAG: workflow made up of tasks with dependencies. Define a DAG in Python: from airflow.models import DAG from datetime import datetime #create a default arguments dictionary (optional) default_arguments = { ...
Download Data using curl https://curl.haxx.se/download.html curl -O https://websitename.com/file001.txt #-O -> download file with it's name curl -o newname.txt https://websitename.com/file001.txt #-o -> download file with new name curl -O...
1. Importing Data from Flat Files and Spreadsheets read_csv for all flat files import pandas as pd data = pd.read_csv('file.csv') data = pd.read_csv('file.tsv', sep='\t') #choose columns to load by name col_names = ['STATEFIPS', 'STATE', 'zipcode', ...