Gradual fetching of data by scrolling down in File Manger Application #166

mahsashadi · 2021-09-04T12:06:48Z

According to this issue, I am implementing gradual fetching by scrolling in FM app. I have two issues.

I need to detect when the whole data is fetched to stop fetching the next time scroll hits the bottom.
So I think I need to initially get total number of directories and files of a path (without calling readdir api and getting the list) and save it into some state.
I checked the stat api, but it does not provide this attribute.
I need to have the last file returned from server, but the list returned here is not in the same order as server response, due to some sorting done here.

mahsashadi · 2021-09-04T12:14:34Z

BTW, What is your idea about adding this pagination feature to your file manager app?
As you suggest here, it can be used by adapters supported this interface in their api.
If you agree, I can make a pull request when it is totally done.

andersevenrud · 2021-09-04T12:15:16Z

Hm.

Relying on either the total number of files or the last one to figure out when to stop does not sound like a good idea.

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

mahsashadi · 2021-09-04T12:23:12Z

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

Yes, seems a better idea, thanks :)

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

andersevenrud · 2021-09-04T12:33:52Z

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

It is possible just to use a page number 🤔

const pageSize = 10
let currentPage = 0
let reachedEnd = false

function changedDirectory() {
  currentPage = 0
  reachedEnd = false
}

async function scrollReachedBottom() {
  if (reachedEnd) return
  const nextPage = currentPage + 1
  const result = await readdir('home:/', { page: nextPage, pageSize })
  reachedEnd = result.length < pageSize
  currentPage = nextPage
}

Now you have a offset on the server: page * pageSize.

mahsashadi · 2021-09-05T08:36:41Z

But it seems my storage pagination API does not support offset.
https://docs.openstack.org/swift/latest/api/pagination.html

andersevenrud · 2021-09-05T13:05:36Z

I see. I'm not really sure how to solve this then without also adding sorting as an option (default true).

mahsashadi · 2021-09-05T13:57:32Z

So do you think you can add this feature?
Or if it is needed to be done by myself, I can start working on that.

andersevenrud · 2021-09-05T14:00:59Z

Feel free to have a look at it. This would also mean that the file manager would have to run the sort manually to make it behave the same.

mahsashadi · 2021-09-05T14:05:51Z

Great :)
Do you have any idea which parts of codes are potential for changing and also where this boolean option should be set.

andersevenrud · 2021-09-05T14:11:44Z

I was just thinking of adding it to the options and add the logic to the VFS function:

osjs-client/src/vfs.js

Line 84 in 385b38f

export const readdir = (adapter, mount) => (path, options = {}) =>

osjs-client/src/vfs.js

Lines 70 to 75 in 385b38f

    
           const handleDirectoryList = (path, options) => result => 
        
             Promise.resolve(result.map(stat => createFileIter(stat))) 
        
               .then(result => transformReaddir(pathToObject(path), result, { 
        
                 showHiddenFiles: options.showHiddenFiles !== false, 
        
                 filter: options.filter 
        
               }));

mahsashadi · 2021-09-05T16:48:52Z

This would also mean that the file manager would have to run the sort manually to make it behave the same.

About this part, when a new page is fetched, I think sorting the accumulative list (previous pages+ new page) leads to a bad UX. The files of previous pages and the new one might be mixed, and the user can not scroll just among files of new page.

andersevenrud · 2021-09-05T17:00:39Z

I was only thinking about applying sorting per result.

Because in adapters that does not support this you'd only get one result.

mahsashadi · 2021-09-06T11:48:07Z

So by adding sortFiles option to this method as below:

// Handles directory listing result(s)
const handleDirectoryList = (path, options) => result =>
  Promise.resolve(result.map(stat => createFileIter(stat)))
    .then(result => transformReaddir(pathToObject(path), result, {
      showHiddenFiles: options.showHiddenFiles !== false,
      filter: options.filter,
      sortFiles: options.sortFiles
    }));

We just need to check sortFiles option in transformReaddir method and do sorting if it is true. This way it works :)

The question is where should I config this boolean option? Maybe it is needed to add this config to src/client/config.js:

  fileManager:{
    pageSize: 1000,
    sortFiles: false
  }

andersevenrud · 2021-09-06T11:52:16Z

The question is where should I config this boolean option?

Doesn't this just fit right into the FM readdir function ?

mahsashadi · 2021-09-06T12:10:26Z

BTW, What is your idea about adding this pagination feature to your file manager app?
As you suggest here, it can be used by adapters supported this interface in their api.
If you agree, I can make a pull request when it is totally done.

So when we complete this feature, are you going to add it to your FM app? If so is it possible to set in readdir as below?

const options = {
  showHiddenFiles: proc.settings.showHiddenFiles,
  sortFiles: false,
  page:{
    size:1000,
    number: state.currentPage,
    lastFetchedFile: state.lastFetchedFile
  }
};

In this way when sorting (deafult is true) is going to happen?

andersevenrud · 2021-09-06T12:12:50Z

If so is it possible to set in readdir as below?

Yes, that will be possible. However, it would be nice to agree on a default option and properties for "paging" so that it can work for all adapters, not just the one that you're making.

mahsashadi · 2021-09-07T11:40:19Z

For sure we should agree on paging options. I've just only used some options for testing the functionality.

Up to now, Three options seem necessary in my mind:

size of each page
number of current page (for storage services that use offset)
last fetched file name (for storage services like our case)

mahsashadi · 2021-09-11T12:24:44Z

Up to now, Three options seem necessary in my mind:

So what is your opinion ?

mahsashadi · 2021-09-19T11:02:05Z

I again find some time to work on paging project :)
So if you have any time, it's good to agree on paging properties.
Thanks a lot.

andersevenrud · 2021-09-19T13:19:40Z

So if you have any time

Sadly, I've been busy with a lot of other things. Those three options are fine, but I can also thing of another one: "pagination token". This is for examplle have in Google Drive. Maybe there are other options there to consider as well.

mahsashadi · 2021-09-28T12:53:31Z

Sometimes I fetch some pages in File manager app, i faced the following error (the page list is returned correctly)

Do you know what can cause this error?

andersevenrud · 2021-09-28T13:20:46Z

Hm. Pretty sure that this can occur if you remove something from the VDOM while it's updating. Does it cause any issues ?

mahsashadi · 2021-09-28T17:23:40Z

It causes new page list items does not show on the file manager app.

andersevenrud · 2021-09-28T17:34:15Z

Can you share the code that causes this ?

mahsashadi · 2021-09-28T17:39:16Z

And there is also another question.
By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

If you agree, what is your suggestion for its implementation?

mahsashadi · 2021-09-28T17:40:04Z

Can you share the code that causes this ?

I do not have access right now. I will share it tomorrow 🙏

andersevenrud · 2021-09-28T17:46:42Z

By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

This would require getting some additional information, which is now not really possible in the VFS 🤔

I'm starting to think maybe having a new scandir endpoint that is designed for doing things like this so that you get a proper server response:

{
  count: number, // Files in this response (page)
  totalFiles: number, // Total files in the directory
  totalPages: number, // Total number of pages
  currentPage: number | string, // An identifier that is used to identify the current page
  next?: {}, // Options for scandir to give next page ?
  prev?: {}, // Options to scandir give previois page ?
  data: VFSFile[]
}

I do not have access right now. I will share it tomorrow pray

Great!

mahsashadi · 2021-09-28T17:59:31Z

How about providing number of files and directories of each dir in stat response.

So we should call stat whenever a new mountpoint or directory is selected in app.

andersevenrud · 2021-09-28T18:03:44Z

Oh, yeah. I didn't think of that 🤦‍♂️ That could work!

mahsashadi · 2021-09-29T11:40:15Z

These are changes in filemanger application index file (till now):

const PAGE_SIZE = 10;

/**
 * Update some state properties when selected directory/file changed
 */
const updateState = (state) => {
  state.currentList = [];
  state.currentPage = 0;
  state.fetchAllPages = false;
  state.currentLastItem = '';
};

/**
 * Detect pagination capability when selected mountpoint changed
 */
const isPagingActive = async (core, path) => {
  const vfs = core.make('osjs/vfs');
  const capabilities = await vfs.capabilities(path);
  return capabilities.pagination;
};

/**
 * Create files list by concating new page items by previous fetched pages items
 */
const createPagesList = async (proc, state, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles,
    page:{
      size:PAGE_SIZE,
      number: state.currentPage,
      marker: state.currentLastItem,
      token: ''
    }
  };

  let list = await vfs.readdir(dir, options);
  // FIXME: the special file `..` should be handled in a better way (not add per page).
  if(list.length === 0 || (list.length === 1 && list[0].filename === '..')) {
    state.fetchAllPages = true;
  }
  if(list.length !== 0 && list !== state.currentList) {
    // FIXME: the special file `..` should be handled in a better way (not add per page).
    if(state.currentList.length !== 0 && list[0].filename === '..' && state.currentList[0].filename === '..') {
      list = list.filter(item => item.filename !== '..');
    }
    state.currentLastItem = list.length !== 0 ? list[list.length - 1].filename : null;
    state.currentList = state.currentList.concat(list);
    list = state.currentList;
  } else {
    list = state.currentList;
  }
  return list;
};

/**
 * Create total list of files when pagination is not supported
 */
const createTotalList = (proc, vfs, dir) => {
  const options = {
    showHiddenFiles: proc.settings.showHiddenFiles
  };
  return vfs.readdir(dir, options);
};

/**
 * VFS action Factory
 */
const vfsActionFactory = (core, proc, win, dialog, state) => {
  // ...
  const readdir = async (dir, history, selectFile) => {
    if (win.getState('loading')) {
      return;
    }
    // if calling by scroll
    if(dir === undefined) {
      dir = state.currentPath;
      state.currentPage += 1;
    }

    try {
      const message = __('LBL_LOADING', dir.path);
      win.setState('loading', true);
      win.emit('filemanager:status', message);
      let list;
      if(state.pagination) {
        list = await createPagesList(proc, state, vfs, dir);
      }else {
        list = await createTotalList(proc, vfs, dir);
      }
    // ...
};

/**
 * Creates a new FileManager user interface
 */
const createApplication = (core, proc, state) => {
   const createActions = (win) => ({
     // ...

    mountview: listView.actions({
      select: async ({data}) => {
        await updateState(state);
        state.pagination = await isPagingActive(core, data.root);
        win.emit('filemanager:navigate', {path: data.root});
      }
    }),

    fileview: listView.actions({
        // ...
        activate: async ({data}) => {
             await updateState(state);
             win.emit(`filemanager:${data.isFile ? 'open' : 'navigate'}`, data);
      },
        scroll: (ev) => {
        if (state.pagination) {
          const el = ev.target;
          if (state.fetchAllPages) {
            return;
          }
          const hitBottom = (el.scrollTop + el.offsetHeight) >= el.scrollHeight;
          if(hitBottom) {
            win.emit('filemanager:navigate');
          }
        }
      }
)}
 // ...
}

/**
 * Creates a new FileManager window
 */
const createWindow = async (core, proc) => {
  let wired;
  const state = {
    currentFile: undefined,
    currentPath: undefined,
    currentList: [],
    currentPage:0,
    fetchAllPages: false,
    currentLastItem:'',
    pagination: false
  };
  const {homePath, initialPath} = createInitialPaths(core, proc);
  state.pagination = await isPagingActive(core, initialPath);
  // ...
}

And I changed transformReaddir of src/utils/vfs.js as below:

const sortedSpecial = createSpecials(path)
   .sort(sorter(sortBy, sortDir))
   .map(modify);

 const sortFiles = files.filter(filterHidden)
   .filter(filter)
   .map(modify);

 return [
   ...sortedSpecial,
   ...sortFiles
 ];

mahsashadi · 2021-09-29T11:59:49Z

When that error accurs:

New page items in current directory are not shown
By transferring between directories/mountpoints, as shown in pic, it is not truly rendering and last files from previous routed directory are shown in current directory (folder14 and folde22 are not in home but shown!)

andersevenrud · 2021-09-29T16:39:41Z

I think I'll have to actually run these changes myself to get a better understanding. Do you have this in a repo ?

mahsashadi · 2021-10-02T12:28:55Z

Pretty sure that this can occur if you remove something from the VDOM while it's updating

I worked more on that.

This error happened whenever there is at least one duplicate in array list. So It can be solved by filtering the array as below:

  const filenameSet = new Set();
  const filteredArr = list.filter((obj) => {
    const isPresentInSet = filenameSet.has(obj.filename);
    filenameSet.add(obj.filename);
    return !isPresentInSet;
  });

mahsashadi · 2021-10-02T13:11:11Z

What is your suggestion for the page-size and number of pages can be shown on filemanager?

Option A: show as much as pages that are fetched ( 👎 not seems as a good idea, scrolling among huge amount of files will be hard)
Option B: only one page can be shown to the user, means that by fetching new page, the previous page items will be deleted (maybe good, it depends on how we set our page-size)
Option C: we can set a limit for number of pages, for example maximum 5 pages items can be listed.
- when page 6 is fetched, page 1 items will be deleted.
- when page 7 is fetched, page 2 items will be deleted.
- and so on ... .

Option C seems better, but its implementation can be a bit complicated, since filesystems work differently (in my case it works by marker not offset)

mahsashadi · 2021-10-17T13:00:02Z

By having pagination, I think it is better to initially show the total count of directories and files of each directory in statusbar ( not increasing by list updating whenever a new page is fetched)

current statusbar:
20 directories, 10 files, 100 bytes total

new statusbar (option 1):
20/100 directories, 10/200 files, 100/10000 bytes total

new statusbar (option 2):
30/300 files & directories, 100/10000 bytes total

Which option do you think is better? if option 1 is needed, is it possible to return count of dir and files seperately in stat?

for option 2, I changed this code block as below:

const createStat = async stat => {
   let count = null;
   if (stat.isDirectory()) {
     const files = await fs.readdir(realPath);
     count = files.length;
   }
   return ({
     isDirectory: stat.isDirectory(),
     isFile: stat.isFile(),
     mime: stat.isFile() ? mime(realPath) : null,
     size: stat.size,
     path: file,
     count: count,
     filename,
     stat
   });
 };

andersevenrud · 2021-10-17T13:07:19Z

Hm. I think maybe it would be best if it simply said x of 100 files, 100000 bytes total. Look kind of confusing with all those stats in there 🤔

is it possible to return count of dir and files seperately in stat?

Well, only if the filesystem that you're working with supports it.

mahsashadi · 2021-10-17T13:12:17Z

x of 100 files, 100000 bytes total

So by files you mean files and directories and x in x bytes total show current loaded bytes (by fetched pages), true?

mahsashadi · 2021-10-17T13:16:17Z

for option 2, I changed this code block as below:

And what should I do for hidden files? Should I also filter them here server side to have total count without hidden files?

andersevenrud · 2021-10-17T13:21:59Z

With x of 100 files, 100000 bytes total I meant that x was the current loaded count (from fetched pages), and everything behind that was the actual total count.

Should I also filter them here server side to have total count without hidden files?

Hidden files should probably only be counted if the option to show them is enabled.

mahsashadi · 2021-10-17T13:28:53Z

With x of 100 files, 100000 bytes total I meant that x was the current loaded count (from fetched pages), and everything behind that was the actual total count.

Actually my question was about the number of total bytes, which I guess you mean there is no need to show current bytes, just show total bytes.

What is your suggestion for the page-size and number of pages can be shown on filemanager?

So do you have any suggestion for this functionality?

andersevenrud · 2021-10-17T13:30:49Z

which I guess you mean there is no need to show current bytes, just show total bytes.

Yes, the total count.

So do you have any suggestion for this functionality?

Does this even need to be exposed ? If it says x of y files that should be enough.

As for adjusting the page size, this is probably better as an adapter setting, and not something that you change in the UI.

mahsashadi · 2021-10-18T13:08:07Z

Does this even need to be exposed ? If it says x of y files that should be enough

I did not understand this. I previously mentioned these three options.

Option A: show as much as pages that are fetched ( -1 not seems as a good idea, scrolling among huge amount of files will be hard)

Option B: only one page can be shown to the user, means that by fetching new page, the previous page items will be deleted (maybe good, it depends on how we set our page-size)

Option C: we can set a limit for number of pages, for example maximum 5 pages items can be listed.

So by considering x of 100 files, 100000 bytes total format for statusbar, I did not understand which option is good in your idea?

andersevenrud · 2021-10-18T13:53:32Z

I did not understand which option is good in your idea?

None of the options.

There is not any need if the user already sees x of 100 files, 100000 bytes total. The user should not really know about how many pages etc there is, only the total count.

mahsashadi · 2021-10-19T08:19:33Z

There is not any need if the user already sees x of 100 files, 100000 bytes total. The user should not really know about how many pages etc there is, only the total count.

I did not mean to show number of pages somewhere, Actually I am thinking about list of page items (files) itself. Consider situation when 4-5 pages with pageSize 1000 fetched. Scrolling among lots of files might not be user-friendly.
I was thinking about listing items in a way that solves this situation (as option A, B, C)

andersevenrud · 2021-10-19T08:23:16Z

Scrolling among lots of files might not be user-friendly.

I don't really think there's a good way to solve this outside maybe doing some kind of categorization/grouping, which would not even be possible with pagination. Option B is very bad UX and option C will probably make it so that the user can't view all of their files, right ?

mahsashadi · 2021-10-19T08:31:44Z

So your opinion is that it is better to show as many pages as fetched, right?

option C will probably make it so that the user can't view all of their files, right ?

yes, maybe previous files fetched again by scrolling up!

andersevenrud · 2021-10-19T08:39:13Z

So your opinion is that it is better to show as many pages as fetched, right?

Yes, show everything.

yes, maybe previous files fetched again by scrolling up!

Hm. I'm not sure how this would be implemented, or how it would "feel" (UX) 🤔

mahsashadi · 2021-11-20T13:13:58Z

Hi :)

What is your suggestion for refreshing list to support actions like delete, paste and rename while pagination is active.

Imagine we have deleted a file within one of our previous fetched pages.

mahsashadi mentioned this issue Nov 7, 2021

Improve unserialization of VFS URL query parameters os-js/osjs-server#56

Closed

mahsashadi mentioned this issue Nov 22, 2021

Adding pagination capability os-js/osjs-filemanager-application#37

Open

andersevenrud mentioned this issue Jul 28, 2022

VFS Capability detection, other extensions os-js/OS.js#804

Open

mahsashadi mentioned this issue Aug 2, 2022

Add totalSize and totalCount values to VFS stats os-js/osjs-server#66

Open

Gradual fetching of data by scrolling down in File Manger Application #166

Gradual fetching of data by scrolling down in File Manger Application #166

Comments

mahsashadi commented Sep 4, 2021

mahsashadi commented Sep 4, 2021

andersevenrud commented Sep 4, 2021

mahsashadi commented Sep 4, 2021

andersevenrud commented Sep 4, 2021 • edited Loading

mahsashadi commented Sep 5, 2021 • edited Loading

andersevenrud commented Sep 5, 2021

mahsashadi commented Sep 5, 2021

andersevenrud commented Sep 5, 2021

mahsashadi commented Sep 5, 2021

andersevenrud commented Sep 5, 2021 • edited Loading

mahsashadi commented Sep 5, 2021

andersevenrud commented Sep 5, 2021 • edited Loading

mahsashadi commented Sep 6, 2021

andersevenrud commented Sep 6, 2021

mahsashadi commented Sep 6, 2021

andersevenrud commented Sep 6, 2021

mahsashadi commented Sep 7, 2021

mahsashadi commented Sep 11, 2021

mahsashadi commented Sep 19, 2021

andersevenrud commented Sep 19, 2021

mahsashadi commented Sep 28, 2021

andersevenrud commented Sep 28, 2021

mahsashadi commented Sep 28, 2021

andersevenrud commented Sep 28, 2021

mahsashadi commented Sep 28, 2021

mahsashadi commented Sep 28, 2021

andersevenrud commented Sep 28, 2021

mahsashadi commented Sep 28, 2021

andersevenrud commented Sep 28, 2021

mahsashadi commented Sep 29, 2021 • edited Loading

mahsashadi commented Sep 29, 2021

andersevenrud commented Sep 29, 2021

mahsashadi commented Oct 2, 2021

mahsashadi commented Oct 2, 2021

mahsashadi commented Oct 17, 2021

andersevenrud commented Oct 17, 2021

mahsashadi commented Oct 17, 2021

mahsashadi commented Oct 17, 2021

andersevenrud commented Oct 17, 2021 • edited Loading

mahsashadi commented Oct 17, 2021

andersevenrud commented Oct 17, 2021

mahsashadi commented Oct 18, 2021

andersevenrud commented Oct 18, 2021

mahsashadi commented Oct 19, 2021

andersevenrud commented Oct 19, 2021

mahsashadi commented Oct 19, 2021

andersevenrud commented Oct 19, 2021

mahsashadi commented Nov 20, 2021

andersevenrud commented Sep 4, 2021 •

edited

Loading

mahsashadi commented Sep 5, 2021 •

edited

Loading

andersevenrud commented Sep 5, 2021 •

edited

Loading

andersevenrud commented Sep 5, 2021 •

edited

Loading

mahsashadi commented Sep 29, 2021 •

edited

Loading

andersevenrud commented Oct 17, 2021 •

edited

Loading