RCA — Staging Security Incident

1. Executive Summary

Classification: Insider Threat via Compromised/Shared Credentials An attacker used a valid @gohighlevel.com Google account to gain unauthorized access to the GHL staging environment.

The attacker connected to the internal Pritunl VPN, accessed the Kubernetes cluster via kubectl exec, retrieved MySQL credentials from pod environment variables, and exfiltrated database schema and row-count metadata from multiple staging databases. A parallel attack directly connected to the courses-mysql-8-x Cloud SQL instance using a personal MySQL user account.

Scope Limitation No evidence of bulk data row exfiltration (SELECT *) was found. The attacker obtained schema structure, table row counts, and data sizes — not raw row data.

2. Incident Timeline

Apr 13, 12:02 UTC

First failed login attempts on courses-mysql-8-x (user root) from Pritunl VPN server 10.160.0.27

Apr 13, 12:21 UTC

root authenticated successfully to db_membership — first confirmed breach

Apr 13, 13:09 UTC

Multiple external IPs probe Pritunl web portal (port 443) — TLS handshake failures

Apr 21, 08:16 UTC

Credential enumeration on production revex-membership-courses DB from localhost — tried membership-read, test, gaurang — all denied

Apr 27, 19:36 UTC

External IP 223.185.134.190 (BSNL India) attempts Pritunl admin panel — TLS timeout

Apr 27, 19:44:13 UTC

cloudsql.instances.connect IAM call to courses-mysql-8-x from 106.200.15.132 (Airtel India)

Apr 27, 19:44:35 UTC

First Access denied for MySQL user sayeed — 4 attempts over 1 minute

Apr 27, ~20:17 UTC

MySQL user sayeed authenticated successfully to db_membership

Apr 27, 20:17–20:37 UTC

Active 20-minute session — SHOW TABLE STATUS FROM db_membership executed, schema and counts exfiltrated. Hacker reported 1,348,116 rows in posts table (confirmed via growth rate analysis)

Apr 27, 20:37 UTC

Session terminated — communication packets dropped / idle timeout

Apr 28–30

Additional root credential attempts on staging courses-mysql-8-x from Pritunl

3. Attack Chain

1

Initial Access via Compromised GHL Account

Attacker obtained or used a valid @gohighlevel.com Google account — shared internally, leaked, or compromised via phishing/session hijack. Pritunl VPN requires Google SSO — no external attacker can bypass this gate.

2

VPN Authentication

Connected to Pritunl VPN using Google SSO (factors: ["google"]). VPN VLAN assigned: 192.168.8.0/22

3

Kubernetes Cluster Access

gcloud container clusters get-credentials web-users-servers-usc1-stg-cluster — VPN VLAN 192.168.8.0/22 is explicitly whitelisted as master authorized network → kubectl access granted automatically without any additional auth

4

Pod Intrusion via kubectl exec

kubectl exec -n staging -it <appengine-crm-contacts-api pod> -- bash. No APISERVER audit logging enabled → this event is completely invisible in GCP logs.

5

Credential Harvesting

printenv | grep -i mysql inside the pod. Retrieved MYSQL_DB_HOST, MYSQL_DB_USER, MYSQL_DB_PASSWORD — plaintext in environment variables — for the highlevel-staging MySQL instance.

6

Database Schema Exfiltration

Connected to MySQL and ran

SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_ROWS, ROUND(DATA_LENGTH/1024/1024) FROM information_schema.TABLES ORDER BY TABLE_ROWS DESC

— returned all databases, all table names, row counts, and data sizes across the entire instance.

7

Direct Cloud SQL Access (Parallel Vector)

Using Cloud SQL Proxy installed on the Pritunl VM + GCP IAM credentials, connected directly to courses-mysql-8-x as MySQL user sayeed. Ran SHOW TABLE STATUS FROM db_membership → exfiltrated membership schema.

4. Root Causes

RC-1 · Primary

GHL Account Compromise

The Pritunl VPN requires a valid @gohighlevel.com Google account with SSO. The attacker had one. This was either a shared internal account, credentials exposed via chat/email/phishing, or a session token hijack. All 27 GHL employees on VPN during the breach window are legitimate employees — this was not an external attacker.

RC-2 · Critical

Pritunl VPN Server Misconfiguration

The Pritunl VPN server (10.160.0.27) had Cloud SQL Proxy installed and running. A database access tool has no business on a network gateway. This created a direct tunnel from VPN → production-equivalent MySQL without going through any application layer or IAM check.

RC-3 · Critical

GKE Cluster Master Authorized Networks

web-users-servers-usc1-stg-cluster: Pritunl VPN VLAN 192.168.8.0/22 explicitly trusted → anyone on VPN can run kubectl.

servers-us-central1-staging-cluster: No master authorized networks configured → completely open K8s API server.

Both clusters have no APISERVER audit logging → kubectl exec events are not captured.

RC-4 · Critical

MySQL Credentials in Pod Environment Variables

Pods mount MYSQL_DB_USERNAME, MYSQL_DB_PASSWORD, MYSQL_DB_HOST as plaintext environment variables. A single kubectl exec + printenv yields full database access to anyone who can exec into the pod.

RC-5 · High

Personal MySQL User Accounts

A MySQL user named sayeed existed on courses-mysql-8-x. Personal named accounts on shared databases are a security anti-pattern — hard to audit, hard to rotate, and persist beyond employment changes.

RC-6 · High

ProxySQL Default Credentials in Public GitHub Repo

The courses-proxysql-api ConfigMap committed to ghl-revex-backend contains admin_credentials = "admin:admin" and stats_credentials = "stats:stats". Any pod in the default namespace can connect to the admin interface and extract the real MySQL root credentials.

RC-7 · High

No Query-Level Logging on Staging Databases

Both affected MySQL instances have general_log = off and log_output = NONE. Slow query logging is enabled in config but writing nowhere. This made it impossible to determine exactly what queries the attacker ran.

RC-8 · High

Databases Open to Internet

wordpress-site-monitoring has 0.0.0.0/0 authorized. ghl-infra has all = 0.0.0.0/0. go-ai-level-database has individual engineer personal IPs whitelisted with a public IP.

5. Data Exposed

courses-mysql-8-x — db_membership

Table	Schema / PII	Rows	Severity
`users`	email, password (hashed), name, authToken, loginCode	372,731	Critical PII
`purchases`	contactId, amount, paymentId, paymentProvider	512,402	Payment Data
`user_purchases`	userId, contactId, productId, validity dates	511,048	Sensitive
`assessment_status`	userId, score, status, submission JSON	588,396	Sensitive
`posts`	title, description, visibility, locationId	1,348,116	Schema Only
`offers`	amount, type, paymentProvider, source	478,474	Schema Only
`categories`	title, productId, dripDays	410,270	Schema Only
`products`	title, description, locationId	345,143	Schema Only
`product_customisations`	instructorName, instructorBio	439,264	Schema Only
`user_product_tracking`	userId, lastLogin, logins	164,460	Sensitive

highlevel-staging MySQL — Multiple Databases

Database	Content	Scale	Severity
`ssl_certs`	Certificates, chains, Keypairs (private key material)	–	Highest
`email_reporting`	Email delivery — contactId, email, open/click/delivered	30M+ rows	Critical PII
`sms_logs_v2`	SMS logs — phone numbers, contactId, delivery	8.9M+ rows	Critical PII
`attribution`	UTM tracking — locationId, contactId, IP, pageUrl, campaign, order amounts	8M+ rows	Sensitive
`bulk_actions`	Campaign execution logs — locationId, userId, actionType	11M+ rows	Sensitive
`workflow`	Workflow execution — contactName, contactEmail, step content	5K rows	Sensitive
`payments`	Company credits — companyId, credits, amounts	Small	Low
`dialogflow`	Bot conversation metadata — companyId, contactId	802 rows	Low

6. What Was NOT Compromised

✓No evidence of bulk SELECT * data exfiltration — no slow queries, no large network egress
✓Attacker obtained schema metadata only (table names, row counts, sizes) — not raw rows
✓Production revex-membership-courses DB: April 21 credential enumeration from localhost fully denied — no successful access
✓ProxySQL admin:admin vulnerability: no evidence of exploitation in available logs
✓Production GCP project (highlevel-backend): no evidence of direct production DB compromise

7. Remediation Actions

Priority 1 — Critical (Do Now)

# 1. Rotate ALL staging MySQL root passwords
gcloud secrets versions add revex-membership-courses-mysqldb-password \
  --data-file=- --project=highlevel-staging

# 2. Drop personal MySQL user accounts from staging
mysql -e "DROP USER IF EXISTS 'sayeed'@'%';"
mysql -e "SELECT user, host FROM mysql.user;"

# 3. Remove Cloud SQL Proxy from Pritunl VM
gcloud compute ssh pritunl --project=highlevel-staging --zone=asia-south1-a
# stop and disable cloud-sql-proxy, remove binary

# 4. Enable API server audit logging
for cluster in servers-us-central1-staging-cluster \
               web-users-servers-usc1-stg-cluster gke-platform-stg-usc1; do
  gcloud container clusters update $cluster \
    --logging=SYSTEM,WORKLOADS,APISERVER \
    --project=highlevel-staging --zone=us-central1
done

# 5. Close internet-open databases
# Remove 0.0.0.0/0 from wordpress-site-monitoring and ghl-infra

# 6. Enable query logging
gcloud sql instances patch courses-mysql-8-x \
  --database-flags=general_log=on,log_output=FILE --project=highlevel-staging
gcloud sql instances patch highlevel-staging \
  --database-flags=general_log=on,log_output=FILE --project=highlevel-staging

Priority 2 — High (This Week)

Action	Owner
Rotate ProxySQL admin credentials away from `admin:admin`	Revex / Infra
Add master authorized networks to `servers-us-central1-staging-cluster` (currently fully open)	Platform / Infra
Move MySQL credentials from pod env vars to mounted K8s Secrets with restricted RBAC	Revex / Infra
Audit all GHL Google accounts for shared/inactive accounts — enforce MFA	IT / Security
Revoke Pritunl VPN access for accounts inactive >30 days	IT / Security
Add NetworkPolicy to restrict pods from reaching `courses-proxysql-api:6032`	Revex
Rotate `ssl_certs` keypairs if production keys are managed in staging schema	Platform

Priority 3 — Medium (This Month)

Action	Owner
Implement `kubectl exec` alerting via APISERVER audit log	Platform / Security
Enable VPC Flow Logs on staging network	Infra
Implement 30-day Secret rotation automation for DB credentials	Platform
Remove all personal IP whitelists from Cloud SQL authorized networks	Infra
Increase `binlog_expire_logs_seconds` to 2592000 (30 days)	DBA
Enable Cloud SQL Data Access audit logging on sensitive instances	DBA / Security

8. Detection Gaps

Gap	Impact	Fix
No APISERVER logging on GKE clusters	`kubectl exec` completely invisible	Enable APISERVER component in cluster logging
`general_log=off` + `log_output=NONE`	Zero MySQL query visibility	Enable both, ship to Cloud Logging
No Cloud SQL data_access audit logging	Can't see secret reads or schema queries	Enable data_access audit logs on all instances
Pritunl OpenVPN logs not shipped to GCP	VPN session IPs/users invisible	Ship `/var/log/openvpn*.log` to Cloud Logging
ProxySQL admin logs on-disk only	Admin interface access undetectable	Configure ProxySQL stdout logging → GCP
Binary log retention only 7 days	Forensics impossible after 1 week	Increase to 30 days minimum

9. Lessons Learned

VPN access ≠ trusted user. The Pritunl VPN being the sole gate to kubectl, Cloud SQL, and infrastructure makes VPN credential compromise catastrophic. Defense-in-depth requires controls beyond VPN — least-privilege kubectl RBAC, secrets manager, query logging.

Shared/exposed company accounts are as dangerous as external attackers. When a GHL Google account is shared or compromised, the attacker gains the full access footprint of an employee — VPN, kubectl, GCP console.

kubectl exec is a privileged operation. The ability to exec into any pod and read plaintext credentials from env vars should be restricted, audited, and alerted on — the same way direct database access is treated.

Default credentials in a public repo are a ticking clock. admin:admin in a ConfigMap committed to GitHub is accessible to every engineer with repo access and exploitable by every pod in the cluster.

log_output=NONE means no logging, even when you think you have it. Having slow query logging "enabled" but writing nowhere is equivalent to no logging. This masked the entire attack at the query layer.

A VPN server should be a dumb gateway — nothing more. A VPN server with Cloud SQL Proxy, kubectl access, and database credentials is a single point of catastrophic failure for the entire staging environment.

Root Cause Analysis — Staging Environment Security Incident