Project
German Job Market Analysis
A data pipeline that collects job listings from Germany, finds technical skills, and shows the results in a dashboard.
View repositoryOverview
I built this project to learn more about the skills requested in the German job market. It collects public LinkedIn job listings, reads their descriptions, and finds the main technical skills and seniority level for each role. The results are stored in PostgreSQL and shown in a Streamlit dashboard.
What I built
- Collected job listings from Germany with Playwright.
- Used Airflow to run the pipeline in the correct order.
- Used an OpenRouter model to find skills and seniority levels.
- Stored the processed job data in PostgreSQL.
- Checked for existing jobs before making model requests.
- Kept failed batches so they could be tried again.
- Ran the project with Docker Compose.
- Built a Streamlit dashboard to explore the results.
Workflow
- Airflow starts the job collection task.
- Playwright searches for public job listings in Germany.
- The listings and descriptions are saved in a JSON file.
- The parser checks whether each job already exists.
- An OpenRouter model finds the skills and seniority level.
- The processed jobs are stored in PostgreSQL.
- Streamlit reads the data and shows the results.
Tools
- Python
- Playwright
- Apache Airflow
- OpenRouter
- PostgreSQL
- Docker Compose
- Streamlit
- Pandas