CloudQuery's Avatar

CloudQuery

@cloudquery.bsky.social

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

18 Followers  |  10 Following  |  597 Posts  |  Joined: 05.09.2024  |  1.7432

Latest posts by cloudquery.bsky.social on Bluesky

Agents made sense for physical data centers. They make zero sense when 70-80% of your resources are managed services.

We built a guide on API-driven discovery: https://www.cloudquery.io/blog/death-of-agent-based-discovery

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 1    πŸ’¬ 0    πŸ“Œ 0

Security model flips too:

Agents: elevated privileges on every host, distributed credentials, 10,000 binaries to patch

APIs: one read-only IAM role, centralized auditing via CloudTrail, revoke in seconds

10.11.2025 15:15 β€” πŸ‘ 1    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Every cloud service has an API. EC2's DescribeInstances. S3's GetBucketPolicy. RDS's DescribeDBInstances.

APIs return 50-100+ config attributes per resource. Zero installation. Zero compute overhead. Just query and parse JSON.

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

The math is brutal at scale:

β†’ 10,000 instances Γ— $4/month = $40K/year in agent overhead
β†’ 2-5% CPU constantly consumed
β†’ 200-500MB memory per instance
β†’ Agents miss short-lived resources that terminate before registration

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Your Kubernetes pod lives 45 seconds. Your RDS database has no OS you can SSH into. 70-80% of AWS services are managed services with nowhere to install an agent.

Yet we're still trying to deploy agents everywhere.

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Your Lambda function runs 200 milliseconds. Agent initialization takes 2-5 seconds.

The function finishes before the agent even starts. You literally cannot install an agent in serverless.

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

We spent 20 years installing agents on servers. Then AWS gave us APIs.

Why are we still installing agents like it's 2005? 🧡

10.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 6    πŸ“Œ 0

Traditional CMDBs were built for servers lasting 3-5 years. That world doesn't exist anymore.

Read the full guide: https://www.cloudquery.io/blog/real-time-cloud-cmdb-ephemeral-infrastructure

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

We put together a guide on building CMDBs that actually work with ephemeral cloud services.

Covers sync strategies, API rate limits, and why the Infrastructure Lake architecture beats proprietary CMDB apps.

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

We've seen this work at 1,000+ AWS accounts with millions of records per sync.

Extract cloud data to PostgreSQL or BigQuery. Query with SQL. Stop pretending infrastructure lives forever.

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

The answer isn't "scan faster." API rate limits make that impossible.

You need tiered sync strategies:
β†’ Critical (IAM, security groups): 15-30 min
β†’ Important (EC2, RDS): hourly
β†’ Everything else: daily

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Here's what that looks like in practice:

β†’ Compromised Lambdas mine crypto for 5 minutes and vanish
β†’ Ephemeral GPU instances rack up $10K bills with zero trace
β†’ Auditors ask for proof from dates between your scans
β†’ Engineers debug "ghost" pods that never appeared

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Traditional CMDBs with 24-hour discovery windows miss ephemeral resources entirely.

A resource that exists for 30 minutes? Little chance it shows up in your daily scan.

AWS spot instances terminate with 2-minute warnings. Lambda functions execute and vanish.

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Your CMDB updates daily. Your containers live 3 minutes. Your Lambda functions live 300 milliseconds.

See the problem? 🧡

07.11.2025 15:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 6    πŸ“Œ 0

Organizations extracting maximum value understand they're implementing a business capability, not deploying a technical solution.

Full breakdown: https://www.cloudquery.io/blog/five-tips-maximum-value-cloud-asset-inventory

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

5/ Plan for continuous improvement and scale

Technology changes. Priorities shift. Cloud environments expand.

Your asset inventory should adapt to organizational change without major re-architecture.

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

4/ Provide actionable intelligence, not just data

When someone discovers an unencrypted database, they should remediate immediatelyβ€”not just report it.

Connect your inventory to build pipelines, alerting systems, and remediation workflows.

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

3/ Prioritize high-impact use cases first

Don't boil the ocean. Find your highest-value problemβ€”upcoming audit, Q4 cost optimization, security gaps.

Solve it completely. Demonstrate clear ROI. Then expand.

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

2/ Engage stakeholders across teams

Your inventory isn't an IT projectβ€”it's a business capability.

Include FinOps, security, compliance, development, and operations as co-owners from day one. Not just users.

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

1/ Business outcomes over technical features

Don't build it because you can. Draw a direct line from every feature to revenue protection, cost savings, or risk reduction.

If you can't explain the business value in one sentence, don't build it.

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

If these questions take more than 30 seconds to answer, your cloud asset inventory needs work.

Here's what we learned from AWS PSA Keegan Marazzi about building asset inventories that actually get used:

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

You manage 4,782 cloud resources across 6 accounts. Can you tell me which S3 buckets are publicly accessible right now? Which IAM roles haven't been used in 90 days? 🧡

05.11.2025 05:59 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 7    πŸ“Œ 0

Traditional CMDBs solved a real problem in 2006. That world doesn't exist anymore.

Infrastructure is code. Resources are ephemeral. APIs provide real-time state.

Stop forcing cloud into 20-year-old models.

Full comparison: https://www.cloudquery.io/blog/cloud-cmdb-vs-traditional-cmdb-2026

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Security incident example: "Find all public-facing servers with SSH open to 0.0.0.0/0"

Traditional CMDB: Run discovery scan (2 hrs), wait for reconciliation (30 min), manual Excel export. Data already outdated.

Cloud CMDB: One SQL query, under a second.

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Data model gap: Traditional Server CI captures ~10 attributes (hostname, IP, OS).

AWS EC2 instance has 50+ attributes (instance type, VPC, security groups, IAM role, tags, EBS volumes, network interfaces).

Traditional CIs miss 80% of what matters in the cloud.

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Cloud CMDB approach:
- Call cloud provider APIs directly
- Get current state in under a second
- Store native resource attributes in SQL
- Query on-demand with standard SQL

Implementation time: hours.

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Traditional CMDB workflow:
- Install agents on every server
- Schedule discovery scans (daily/hourly)
- Reconcile duplicates
- Force resources into CI templates
- Data is 12-24 hours stale

Implementation time: 2-3 months minimum.

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Gartner reports 70-80% of traditional CMDB projects fail to deliver value.

The reason: agent-based discovery, scheduled scans, and ITIL Configuration Items designed for physical servers can't handle ephemeral cloud infrastructure.

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

Traditional CMDBs were built for servers with names like "web-prod-01" that run for years.

In 2026, that EC2 instance running your Lambda cold start lives for 45 seconds.

Traditional CMDB discovery would schedule a scan for tomorrow. By then, it's gone. 🧡

03.11.2025 14:15 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 6    πŸ“Œ 0

Challenge 3: Real-time security

Security asks for public S3 buckets at 9:30 AM. CMDB last scan ran at 3:00 AM. You schedule new discovery. Wait 2 hours. Export. Filter manually.

Results at 12:15 PM. But 200 new buckets created since 9:30 AM aren't in your report.

01.11.2025 14:00 β€” πŸ‘ 0    πŸ” 0    πŸ’¬ 0    πŸ“Œ 0

@cloudquery is following 10 prominent accounts