The Flextender scraper is a service that extracts job listings from the Flextender platform using authenticated HTTP requests. Since Flextender doesn't provide a public API, we use HTML parsing with JSoup to extract job data.

Architecture

Technology Stack

HTTP Client: OkHttp for making authenticated requests
HTML Parser: JSoup for parsing server-rendered HTML
Authentication: Cookie-based session authentication
Tracking: CrawlJob entity for monitoring scraping progress

How It Works

Makes HTTP request to Flextender jobs listing page with authentication cookies
Extracts job IDs from listing page HTML using CSS selectors
Fetches each job detail page individually
Parses job information (title, company, region, description, etc.)
Creates Job entities in database with proper attributes
Tracks progress and errors via CrawlJob entity

Configuration

Step 1: Obtain Authentication Cookies

You need to manually extract cookies from an authenticated Flextender browser session.

Using Chrome DevTools:

Open Chrome and navigate to https://app.flextender.nl
Log in with your Flextender account
Navigate to /supplier/jobs/recommended
Open Chrome DevTools (F12 or Cmd+Option+I)
Go to Network tab
Refresh the page
Click on any request to app.flextender.nl
Find the Request Headers section
Copy the entire Cookie header value

Example Cookie format:

_flextender_session=eyJhbGci...; remember_user_token=W1sz...

Step 2: Configure Application

Add your cookies to the application configuration:

Using Environment Variables (Recommended for production):

export FLEXTENDER_ENABLED=true
export FLEXTENDER_COOKIES="_flextender_session=eyJhbGci...; remember_user_token=W1sz..."

Using application.yml (For development):

flextender:
  enabled: true
  cookies: "_flextender_session=eyJhbGci...; remember_user_token=W1sz..."

Using .env file (If supported):

FLEXTENDER_ENABLED=true
FLEXTENDER_COOKIES=_flextender_session=eyJhbGci...; remember_user_token=W1sz...

Step 3: Restart Application

Restart the backend application to load the new configuration:

./mvnw spring-boot:run

Usage

Manually Trigger Scraping

Use the REST API endpoint to trigger scraping:

curl -X POST http://localhost:8080/api/crawler/trigger-flextender \
  -H "Content-Type: application/json"

Success Response:

{
  "success": true,
  "message": "Flextender scraping triggered successfully"
}

Error Response (Disabled):

{
  "success": false,
  "error": "Flextender scraping is disabled. Set flextender.enabled=true to enable."
}

Error Response (No Cookies):

{
  "success": false,
  "error": "Flextender cookies not configured. Set flextender.cookies in application.yml"
}

Check Scraping Status

Monitor the logs for scraping progress:

tail -f logs/application.log | grep Flextender

Expected Log Output:

INFO  n.g.s.FlextenderScraperService - Starting Flextender job scraping...
INFO  n.g.s.FlextenderScraperService - Found 15 job listings on Flextender
INFO  n.g.s.FlextenderScraperService - Created job: https://app.flextender.nl/supplier/jobs/view/12345 - Senior Java Developer
INFO  n.g.s.FlextenderScraperService - Flextender scraping completed. 15 jobs processed, 12 new jobs created

View Crawl History

Check the crawl history in the database:

SELECT * FROM crawl_jobs
WHERE crawl_type = 'FLEXTENDER_JOBS'
ORDER BY created_at DESC
LIMIT 10;

Or use the API endpoint:

curl http://localhost:8080/api/crawler/history?page=0&size=10

Job Data Extraction

Extracted Fields

The scraper extracts the following job attributes from Flextender:

| Flextender Field | Job Entity Field | Description | |------------------|------------------|-------------| | .flx-jobsummary-title | title | Job title | | .flx-jobsummary-client | company | Client/company name | | Regio (caption-field) | region | Region (Zuid-Holland, etc.) | | Plaats (caption-field) | city | City name | | Duur (caption-field) | description | Contract duration | | Uren per week (caption-field) | description | Hours per week | | Start (caption-field) | description | Start date | | Verloopt op (caption-field) | description | Application deadline | | Aanvraagnummer (caption-field) | description | Request number | | .css-formattedjobdescription | description | Full job description |

Job Attributes

Created jobs have the following default attributes:

jobType: CONTRACT (default for Flextender jobs)
category: Consultancy (default)
experienceLevel: SENIOR (default)
sourceType: FLEXTENDER (for tracking)
sourceUrl: Full URL to original job posting
active: true (automatically published)

Browser Header Simulation

The scraper includes comprehensive browser headers to avoid detection:

.header("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8")
.header("Accept-Language", "en-US,en;q=0.9,nl;q=0.8")
.header("Accept-Encoding", "gzip, deflate, br")
.header("Referer", "https://app.flextender.nl/")
.header("Sec-Fetch-Dest", "document")
.header("Sec-Fetch-Mode", "navigate")
.header("Sec-Fetch-Site", "same-origin")
.header("Sec-Fetch-User", "?1")
.header("Upgrade-Insecure-Requests", "1")
.header("Connection", "keep-alive")

These headers make requests indistinguishable from real Chrome browser requests.

Rate Limiting & Performance

Built-in Rate Limiting

1 second delay between individual job page requests
Prevents overwhelming Flextender servers
Reduces risk of IP blocking or account suspension

Performance Metrics

Listing page: ~1-2 seconds
Individual job page: ~0.5-1 seconds
Total time for 20 jobs: ~20-25 seconds

Recommended Scraping Frequency

Manual scraping: As needed
Scheduled scraping: 2-4 times per day
High-frequency: Not recommended (risk of blocking)

Troubleshooting

Issue: "Flextender scraping is disabled"

Cause: flextender.enabled is set to false or not configured

Solution:

flextender:
  enabled: true

Issue: "Flextender cookies not configured"

Cause: flextender.cookies is empty or not set

Solution: Follow Step 1 (Obtain Authentication Cookies) above

Issue: "Failed to fetch page - Status: 401"

Cause: Cookies have expired or are invalid

Solution:

Log in to Flextender again
Extract fresh cookies from browser
Update configuration
Restart application

Issue: "Failed to fetch page - Status: 403"

Cause: Flextender detected automated access or IP is blocked

Solution:

Wait 1-2 hours before trying again
Ensure cookies are from the same IP address
Check if rate limiting is working (1 second delay)
Consider using a different IP or network

Issue: "Found 0 job listings"

Possible Causes:

Flextender HTML structure changed
Cookies don't have access to jobs page
CSS selectors are outdated

Solution:

Manually visit https://app.flextender.nl/supplier/jobs/recommended
Verify you can see jobs
Check browser console for errors
Update CSS selectors in FlextenderScraperService.java if needed

Issue: Jobs are created but missing data

Cause: Caption fields in Flextender HTML changed

Solution: Update extractCaptionField() method or CSS selectors in FlextenderScraperService.java

Security Considerations

Cookie Security

⚠️ Never commit cookies to Git
✅ Use environment variables for production
✅ Rotate cookies regularly (every 7-14 days)
✅ Use dedicated Flextender account for scraping

Legal & Ethical

✅ Respect Flextender's Terms of Service
✅ Implement reasonable rate limiting
✅ Only scrape jobs you have access to
⚠️ Check if Flextender provides an official API first

Account Safety

✅ Use a dedicated scraping account (not your main account)
✅ Monitor for unusual login notifications
✅ Implement error handling to stop on repeated failures
⚠️ Don't scrape too frequently (risk of account suspension)

Future Enhancements

Planned Improvements

Scheduled Scraping: Automatic scraping via cron job or scheduler
Cookie Refresh: Automatic cookie renewal mechanism
Proxy Support: Rotate IPs to avoid rate limiting
Job Updates: Detect and update changed jobs
Duplicate Detection: More sophisticated duplicate handling
Category Mapping: Intelligent job category detection
Experience Level Parsing: Extract seniority from job description

Potential Issues

Cookie Expiration: Cookies typically expire after 7-30 days
HTML Changes: Flextender may change HTML structure
CAPTCHA: Flextender may add CAPTCHA for automated access
IP Blocking: Excessive scraping may result in IP blocks

Support

For issues or questions:

Check logs for detailed error messages
Verify cookies are still valid
Check Flextender website for changes
Contact GloryLabs support

Last Updated: 2025-10-14 Version: 1.0 Author: Claude Code Status: Production Ready

Reacties

Nog geen reacties

Athena — mahmoud-consultancy/archive/old-docs/setup/FLEXTENDER_SCRAPER_SETUP.md

title: Flextender Scraper Setup Guide date: 2025-10-14 status: Active tags: [flextender, scraping, configuration, cookies, authentication]

Flextender Scraper Setup Guide

Overview

Architecture

Technology Stack

How It Works

Configuration

Step 1: Obtain Authentication Cookies

Using Chrome DevTools:

Step 2: Configure Application

Step 3: Restart Application

Usage

Manually Trigger Scraping

Check Scraping Status

View Crawl History

Job Data Extraction

Extracted Fields

Job Attributes

Browser Header Simulation

Rate Limiting & Performance

Built-in Rate Limiting

Performance Metrics

Recommended Scraping Frequency

Troubleshooting

Issue: "Flextender scraping is disabled"

Issue: "Flextender cookies not configured"

Issue: "Failed to fetch page - Status: 401"

Issue: "Failed to fetch page - Status: 403"

Issue: "Found 0 job listings"

Issue: Jobs are created but missing data

Security Considerations

Cookie Security

Legal & Ethical

Account Safety

Future Enhancements

Planned Improvements

Potential Issues

Related Documentation

Support

Reacties