lacucinadiadine

I'm currently working with Spark 2.1 and have a main script that calls a helper module that con… Read more Pyspark 2.1: Importing Module With Udf's Breaks Hive Connectivity

I'm loading many versions of JSON files to spark DataFrame. some of the files holds columns A,B… Read more Spark - Set Null When Column Not Exist In Dataframe

I have a script with the below setup. I am using: 1) Spark dataframes to pull data in 2) Converting… Read more Convert A Pandas Dataframe To A Pyspark Dataframe

I want to be able to use a Scala function as a UDF in PySpark package com.test object ScalaPySpark… Read more How To Use Scala Udf In Pyspark?

I am writing a function which takes a RDD as input splits the comma separated values then convert… Read more Error In Labelled Point Object Pyspark

I have a dataframe named result in pyspark and I want to apply a udf to create a new column as belo… Read more Apply Udf To Multiple Columns And Use Numpy Operations

I want to use some string similarity functions that are not native to pyspark such as the jaro and … Read more Pyspark: How To Deal With Null Values In Python User Defined Functions

I have a spark dataframe (prof_student_df) that lists student/professor pair for a timestamp. There… Read more Implementing A Recursive Algorithm In Pyspark To Find Pairings Within A Dataframe

I have a query about loading the schema onto cdsw using pyspark. I have a dataframe which is create… Read more Uploading Custom Schema From A Csv File Using Pyspark

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried t… Read more How Do I Add A New Column To A Spark Dataframe (using Pyspark)?

How to parse JSON string of nested lists to spark data frame in pyspark ? Input data frame: +------… Read more Spark: How To Parse Json String Of Nested Lists To Spark Data Frame?

Pyspark: Split and select part of the string column values How can I select the characters or file … Read more Pyspark: Select Part Of The String(file Path) Column Values

I am using PySpark. I have a column ('dt') in a dataframe ('canon_evt') that this a… Read more Pyspark 1.5 How To Truncate Timestamp To Nearest Minute From Seconds

I have a PySpark DataFrame, df1, that looks like: CustomerID CustomerValue CustomerValue2 12 … Read more Cosine Similarity For Two Pyspark Dataframes

I have 2 dataframes A(35 Million records) and B(30000 records) A |Text | ------- | pqr | ------- |… Read more Pyspark Crossjoin Between 2 Dataframes With Millions Of Records