Skip to content

[7257] Add proper pagination to Hosted and Remote Instances pages + N+1 fix for Hosted Instances #7353

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
n-lark wants to merge 17 commits into
base: main
Choose a base branch
from
Open

[7257] Add proper pagination to Hosted and Remote Instances pages + N+1 fix for Hosted Instances #7353

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
548ec55
Add server-side pagination, search, and sort to Hosted Instances and …
n-lark May 27, 2026
db3cbbe
Add loading overlay to prevent table looking glitchy on page change
n-lark May 27, 2026
d4dba5e
Fix dashboard only users seeing pagination footer, update devices lis…
n-lark May 27, 2026
8c9e889
Add Cypress test coverage for pagination flows; update spec intercept…
n-lark May 28, 2026
1fc84e9
Fix dashboard role status reads and refetch device list on delete to …
n-lark May 28, 2026
ff8f0cd
Merge branch 'main' into 7257-pagination
cstns Jun 2, 2026
93e2ca8
Refactor /teams/:teamId/projects schema to use : 'PaginationParams' v…
n-lark Jun 2, 2026
c8eca28
Remove loading overlay from devices and instances
n-lark Jun 2, 2026
ed63ccf
Merge branch 'main' into 7257-pagination
cstns Jun 3, 2026
35ab9dc
Address PR feedback: typed Pagination.vue, prune dead nextCursor + fi…
n-lark Jun 3, 2026
10120d0
PR feedback: pagination self-hides when total <= pageSize
n-lark Jun 3, 2026
8372ee7
PR feedback: collapse byTeam/getInstances args into a pagination object
n-lark Jun 3, 2026
07b93f0
Restore ?limit handling for unpaginated /projects requests
n-lark Jun 3, 2026
2c346e6
Add server-side sort for deviceGroup.name on device list
n-lark Jun 3, 2026
23e2812
Drop hasLoadedOnce flag — derive first-paint from data presence
n-lark Jun 3, 2026
b558e47
Merge remote-tracking branch 'origin/main' into 7257-pagination
n-lark Jun 3, 2026
b5cbd25
Fix old heroicons import
n-lark Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .eslintrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@
"commonjs": false
},
"parserOptions": {
"parser": {
"ts": "@typescript-eslint/parser"
},
"ecmaVersion": 2022,
"sourceType": "module"
},
Expand Down
3 changes: 2 additions & 1 deletion forge/db/controllers/Device.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,9 @@ module.exports = {
}

if (paginationOptions.sort) {
// Prefer `dir`; fall back to legacy `order` query param if a caller still sends it.
paginationOptions.order = {
[paginationOptions.sort]: paginationOptions.order
[paginationOptions.sort]: paginationOptions.dir || paginationOptions.order
}
delete paginationOptions.sort
delete paginationOptions.dir
Expand Down
18 changes: 18 additions & 0 deletions forge/db/controllers/Project.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,24 @@ module.exports = {
app.caches.createCache(inflightDeploys)
},

getProjectPaginationOptions: function (app, request) {
const { page, limit, sort, dir } = request.query || {}
const pagination = {}
if (typeof page === 'number') {
const pageSize = limit || 25
pagination.page = page
pagination.limit = pageSize
pagination.offset = (page - 1) * pageSize
} else if (limit) {
pagination.limit = limit
}
if (sort) {
pagination.sort = sort
pagination.dir = dir || 'asc'
}
return pagination
},

/**
* Get the in-flight state of a project
* @param {*} app
Expand Down
52 changes: 40 additions & 12 deletions forge/db/models/Device.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,13 @@ module.exports = {
} = {}) => {
// Pagination
const limit = Math.min(parseInt(pagination.limit) || 100, 100)
if (pagination.cursor) {
const pageNum = pagination.page ? parseInt(pagination.page) : null
const usingOffset = pageNum && pageNum >= 1
const offset = usingOffset ? (pageNum - 1) * limit : null
if (usingOffset) {
// Offset mode and cursor mode are mutually exclusive — drop any incoming cursor.
delete pagination.cursor
} else if (pagination.cursor) {
const cursors = pagination.cursor.split(',')
cursors[cursors.length - 1] = M.Device.decodeHashid(cursors[cursors.length - 1])
pagination.cursor = cursors.join(',')
Expand Down Expand Up @@ -535,6 +541,8 @@ module.exports = {
}
} else if (key === 'instance') {
order.unshift([M.Project, 'name', pagination.order[key] || 'ASC'])
} else if (key === 'deviceGroup.name') {
order.unshift([M.DeviceGroup, 'name', pagination.order[key] || 'ASC'])
} else if (key === 'state-priority') {
order.unshift([literal(`
CASE
Expand Down Expand Up @@ -627,18 +635,31 @@ module.exports = {
}
const statusOnlyIncludes = projectInclude.include?.where ? [projectInclude] : []

const findAllOptions = {
where: buildPaginationSearchClause(pagination, where, ['Device.name', 'Device.type'], {}, order),
include: pagination.statusOnly ? statusOnlyIncludes : includes,
order,
limit: pagination.statusOnly ? null : limit
}
if (usingOffset && !pagination.statusOnly) {
findAllOptions.offset = offset
}
// Mirror findAll's search/filter so total matches; strip cursor so we count the full match, not rows after it.
const countWhere = buildPaginationSearchClause(
{ ...pagination, cursor: null },
where,
['Device.name', 'Device.type'],
{},
order
)
const [rows, count] = await Promise.all([
this.findAll({
where: buildPaginationSearchClause(pagination, where, ['Device.name', 'Device.type'], {}, order),
include: pagination.statusOnly ? statusOnlyIncludes : includes,
order,
limit: pagination.statusOnly ? null : limit
}),
this.count({ where, include: statusOnlyIncludes })
this.findAll(findAllOptions),
this.count({ where: countWhere, include: statusOnlyIncludes })
])

let nextCursors = []
if (rows.length === limit && limit > 0) {
// Cursor-based forward continuation only makes sense in cursor mode.
if (!usingOffset && rows.length === limit && limit > 0) {
const lastRow = rows[rows.length - 1]
nextCursors = order.map((sortProps) => {
// Model, key, dir
Expand Down Expand Up @@ -666,10 +687,17 @@ module.exports = {
})
}

const meta = {
next_cursor: nextCursors.length > 0 ? nextCursors.join(',') : undefined
}
if (usingOffset) {
meta.page = pageNum
meta.pageSize = limit
meta.total = count
meta.pageCount = Math.max(1, Math.ceil(count / limit))
}
return {
meta: {
next_cursor: nextCursors.length > 0 ? nextCursors.join(',') : undefined
},
meta,
count,
devices: rows
}
Expand Down
47 changes: 45 additions & 2 deletions forge/db/models/Project.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,15 +552,23 @@ module.exports = {
})
},
byTeam: async (teamIdOrHash, {
pagination = null,
query = null,
instanceId = null,
includeAssociations = true,
includeSettings = false,
includeMeta = false,
limit = null,
orderByMostRecentFlows = false,
excludeApplications = null
} = {}) => {
const {
page = null,
limit = null,
offset = null,
sort = null,
dir = 'asc'
} = pagination || {}
const withTotal = page !== null
let teamId = teamIdOrHash
if (typeof teamId === 'string') {
teamId = M.Team.decodeHashid(teamId)
Expand Down Expand Up @@ -613,8 +621,31 @@ module.exports = {
if (limit !== null) {
queryObject.limit = limit
}
if (offset !== null) {
queryObject.offset = offset
}

if (includeMeta && orderByMostRecentFlows) {
const direction = String(dir).toLowerCase() === 'desc' ? 'DESC' : 'ASC'
if (sort) {
const sortMap = {
name: ['name'],
createdAt: ['createdAt'],
updatedAt: ['updatedAt'],
'application.name': [{ model: M.Application }, 'name'],
flowLastUpdatedAt: [{ model: M.StorageFlow }, 'updatedAt']
}
const mapped = sortMap[sort]
if (mapped) {
// flowLastUpdatedAt needs the StorageFlow include — only valid with includeMeta
if (sort === 'flowLastUpdatedAt' && !includeMeta) {
// fall through to default ordering below
} else {
queryObject.order = [[...mapped, `${direction}${sort === 'flowLastUpdatedAt' ? ' NULLS LAST' : ''}`]]
}
}
}

if (!queryObject.order && includeMeta && orderByMostRecentFlows) {
queryObject.order = [
[literal(`
CASE
Expand Down Expand Up @@ -647,6 +678,18 @@ module.exports = {
}
}

if (withTotal) {
const countInclude = [{
model: M.Team,
where: { id: teamId },
attributes: []
}]
const [rows, total] = await Promise.all([
this.findAll(queryObject),
this.count({ where: queryObject.where, include: countInclude, distinct: true, col: 'id' })
])
return { rows, total }
}
return this.findAll(queryObject)
},
getProjectTeamId: async (id) => {
Expand Down
12 changes: 10 additions & 2 deletions forge/routes/api-docs.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,11 @@ module.exports = fp(async function (app, opts) {
properties: {
query: { type: 'string' },
cursor: { type: 'string' },
limit: { type: 'number' }
limit: { type: 'number' },
page: { type: 'number', minimum: 1 },
sort: { type: 'string' },
dir: { type: 'string', enum: ['asc', 'desc'] },
order: { type: 'string', enum: ['asc', 'desc'] }
}
})

Expand All @@ -159,7 +163,11 @@ module.exports = fp(async function (app, opts) {
type: 'object',
properties: {
next_cursor: { type: 'string' },
previous_cursor: { type: 'string' }
previous_cursor: { type: 'string' },
page: { type: 'number' },
pageSize: { type: 'number' },
total: { type: 'number' },
pageCount: { type: 'number' }
}
})
app.addSchema({
Expand Down
1 change: 1 addition & 0 deletions forge/routes/api/projectDevices.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ module.exports = async function (app) {
schema: {
summary: 'Get a list of devices assigned to an instance',
tags: ['Instances'],
query: { $ref: 'PaginationParams' },
params: {
type: 'object',
properties: {
Expand Down
58 changes: 36 additions & 22 deletions forge/routes/api/team.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,30 +365,32 @@ module.exports = async function (app) {
*/
app.get('/:teamId/projects', {
preHandler: app.needsPermission('team:projects:list'),
query: {
type: 'object',
properties: {
limit: {
type: 'number',
nullable: true
},
includeMeta: {
type: 'boolean',
nullable: true,
default: false
},
orderByMostRecentFlows: {
type: 'boolean',
nullable: true,
default: false
}
schema: {
query: {
allOf: [
{ $ref: 'PaginationParams' },
{
type: 'object',
properties: {
sort: {
type: 'string',
enum: ['name', 'createdAt', 'updatedAt', 'application.name', 'flowLastUpdatedAt']
Copy link
Copy Markdown
Contributor

@cstns cstns Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pagination params already include sort as an option, was this only so we can type the endpoint more tightly?

Copy link
Copy Markdown
Contributor Author

@n-lark n-lark Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah PaginationParams.sort is just a string, so unknown values would silently fall through to the default order. The enum makes the route 400 instead.

},
includeMeta: { type: 'boolean', default: false },
orderByMostRecentFlows: { type: 'boolean', default: false }
}
}
]
}
}
}, async (request, reply) => {
const includeMeta = request.query.includeMeta
const pagination = app.db.controllers.Project.getProjectPaginationOptions(request)

const options = {
includeSettings: true,
limit: request.query.limit,
pagination,
query: request.query.query?.trim() || null,
includeMeta,
orderByMostRecentFlows: request.query.orderByMostRecentFlows
}
Expand All @@ -406,7 +408,10 @@ module.exports = async function (app) {
}
}

const projects = await app.db.models.Project.byTeam(request.params.teamId, options)
const queryResult = await app.db.models.Project.byTeam(request.params.teamId, options)
Copy link
Copy Markdown
Contributor

@cstns cstns Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is where we could send the pagination param in

Copy link
Copy Markdown
Contributor Author

@n-lark n-lark Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup refactored.

const paginated = pagination.page != null
const projects = paginated ? queryResult.rows : queryResult
const total = paginated ? queryResult.total : null

if (projects) {
let result = await app.db.views.Project.instancesList(projects, {
Expand All @@ -420,10 +425,19 @@ module.exports = async function (app) {
return { id: e.id, name: e.name }
})
}
reply.send({
count: result.length,
const response = {
count: paginated ? total : result.length,
projects: result
})
}
if (paginated) {
response.meta = {
page: pagination.page,
pageSize: pagination.limit,
total,
pageCount: Math.max(1, Math.ceil(total / pagination.limit))
}
}
reply.send(response)
} else {
reply.code(404).send({ code: 'not_found', error: 'Not Found' })
}
Expand Down
1 change: 0 additions & 1 deletion forge/routes/api/teamDevices.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,6 @@ module.exports = async function (app) {
}
const devices = await app.db.models.Device.getAll(paginationOptions, where, options)
devices.devices = devices.devices.map(d => app.db.views.Device.device(d, { statusOnly: paginationOptions.statusOnly }))
devices.count = devices.devices.length
reply.send(devices)
Comment thread
n-lark marked this conversation as resolved.
})

Expand Down
26 changes: 23 additions & 3 deletions frontend/src/api/team.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,19 +196,39 @@ const getTeamInstancesList = async (teamId) => {
}

const getInstances = async (teamId, {
limit = 20,
pagination = null,
includeMeta = false,
orderByMostRecentFlows = false
} = {}) => {
const {
page = null,
limit = 20,
query = null,
sort = null,
dir = null
} = pagination || {}
const params = new URLSearchParams()

params.append('limit', limit.toString())

if (page !== null) params.append('page', String(page))
if (query) params.append('query', query)
if (sort) params.append('sort', sort)
if (dir) params.append('dir', dir)
if (includeMeta) params.append('includeMeta', includeMeta.toString())
if (orderByMostRecentFlows) params.append('orderByMostRecentFlows', orderByMostRecentFlows.toString())

return await client.get(`/api/v1/teams/${teamId}/projects?${params.toString()}`)
.then(res => res.data)
const res = await client.get(`/api/v1/teams/${teamId}/projects?${params.toString()}`)
res.data.projects = res.data.projects.map(r => {
Copy link
Copy Markdown
Contributor

@cstns cstns Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what changed that we need to do this now, and we didn't before? nothing stood out for me

Copy link
Copy Markdown
Contributor Author

@n-lark n-lark Jun 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So when I tested with a dashboard-only user I got a forever spinner. The old code path called getInstanceMeta() per-row after the initial fetch, which set meta.state on every row including dashboard ones.

The N+1 fix removed that fan-out, so dashboard rows (whose endpoint doesn't return meta) lost their meta.state. Swapped the column mapping from meta.state to status and added this normalization to copy meta.state onto status so both shapes now read the same field.

if (r.flowLastUpdatedAt) {
r.flowLastUpdatedSince = daysSince(r.flowLastUpdatedAt)
}
if (r.meta?.state) {
r.status = r.meta.state
}
return r
})
return res.data
}

const getTeamMembers = (teamId) => {
Expand Down
Loading
Loading