Welcome!
  Post Categories
search 1 data science 120 data engineering 181 machine learning 6 optimization 3 Metaverse 1 physics 3 stack overflow 1 image processing 3 algorithm 13 VR 1 fintech 7 web3 1
Difference between 'python -m pip install' and 'pip install' Difference between 'python -m pip install' and 'pip install'
Installing packages in Python can be done using the package manager pip. There are two ways to run pip to install packag
2023-02-08 data engineering
python
Logging metrics and tags of multiple models into one mlflow in Databricks Logging metrics and tags of multiple models into one mlflow in Databricks
There are situations where we need to run multiple notebooks or algorithms that are closely related. In this case, it ma
2023-02-07 data engineering
databricks
Spark SQL - Collecting Columns into Lists after Groupby Spark SQL - Collecting Columns into Lists after Groupby
In Spark SQL, you may want to collect the values of one or more columns into lists after grouping the data by one or mor
2023-02-01 data engineering
spark pyspark
In PySpark how to filter dataframe column using unique values from another dataframe In PySpark how to filter dataframe column using unique values from another dataframe
Here is one common task in PySpark: how to filter one dataframe column are from unique values from anther dataframe? Met
2023-01-22 data engineering
spark pyspark
How to generate access token in Databricks How to generate access token in Databricks
In Databricks, personal access tokens can be used for secure authentication to the Databricks API instead of passwords.
2023-01-20 data engineering
databricks
How to fix google cloud API user rate limit exceeded error How to fix google cloud API user rate limit exceeded error
We are using the google cloud translate API, and suddenly noticed it gives error.A closer check of the logs shows there
2022-11-19 data engineering
google cloud
Create secrets scopes and keys for databricks, and how to give scope access to someone else Create secrets scopes and keys for databricks, and how to give scope access to someone else
In previous blogs, we have discussed about how to install the databricks command line tool (CLI),and how to add authenti
2022-10-16 data engineering
databricks
How to authenticate databricks command line tool How to authenticate databricks command line tool
Once we have installed the databricks command line tool (CLI), we still need to add authentication to the databricks web
2022-10-12 data engineering
databricks
How to install databricks command line tool in linux and windows How to install databricks command line tool in linux and windows
Sometimes we need to interact with databricks command line to get work done, for example, create new secrets scope and k
2022-10-11 data engineering
databricks
How to make AWS S3 buckets public through bucket policy How to make AWS S3 buckets public through bucket policy
The first step is to create a S3 buckets, here is how we can create one on AWS website through UI: After clicking the “c
2022-10-01 data engineering
aws s3
How to auto run python jobs using crontab How to auto run python jobs using crontab
In ubuntu Linux，many times we need to run certain tasks regularly, such as every 10 minutes, at 2 o’clock in the morning
2021-10-24 data engineering
crontab
How to serve the Flask app with nginx as the reverse proxy How to serve the Flask app with nginx as the reverse proxy
We are going to build a simple and smooth process using the following steps: if you don’t have uwsgi installed, try th
2021-10-01 data engineering
flask app nginx
15 / 16